Jul 19 16:36:24.497952 ip-10-0-142-78 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 19 16:36:24.498091 ip-10-0-142-78 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 19 16:36:24.498158 ip-10-0-142-78 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 19 16:36:24.498512 ip-10-0-142-78 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 19 16:36:34.689624 ip-10-0-142-78 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 19 16:36:34.689643 ip-10-0-142-78 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c5d555dd154c482ca903def9e29df93a --
Jul 19 16:38:08.192045 ip-10-0-142-78 systemd[1]: Starting Kubernetes Kubelet...
Jul 19 16:38:08.640138 ip-10-0-142-78 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 19 16:38:08.640138 ip-10-0-142-78 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 19 16:38:08.640138 ip-10-0-142-78 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 19 16:38:08.640138 ip-10-0-142-78 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 19 16:38:08.640138 ip-10-0-142-78 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 19 16:38:08.643173 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.643081    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 19 16:38:08.651078 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651057    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 16:38:08.651078 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651075    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 16:38:08.651078 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651080    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651083    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651088    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651091    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651093    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651096    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651099    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651102    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651104    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651107    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651111    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651116    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651120    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651123    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651126    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651135    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651139    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651142    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651146    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 16:38:08.651190 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651149    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651152    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651155    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651158    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651160    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651163    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651167    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651170    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651173    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651175    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651178    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651180    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651183    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651186    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651188    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651192    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651195    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651198    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651201    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651204    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 16:38:08.651674 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651206    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651209    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651211    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651214    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651217    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651219    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651222    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651224    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651227    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651230    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651233    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651235    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651238    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651240    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651243    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651246    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651248    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651250    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651253    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651255    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 16:38:08.652403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651258    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651260    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651262    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651266    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651269    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651271    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651273    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651277    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651280    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651282    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651285    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651288    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651290    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651293    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651295    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651298    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651300    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651303    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651305    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 16:38:08.652944 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651308    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651310    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651313    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651316    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651319    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651321    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651324    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651326    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651329    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651331    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651334    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651337    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651339    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651342    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651344    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651346    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651349    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651351    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 16:38:08.653537 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.651354    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 16:38:08.654220 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654204    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 16:38:08.654220 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654220    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654225    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654229    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654232    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654235    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654239    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654242    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654245    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654247    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654250    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654253    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654256    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654258    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654261    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654263    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654266    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654268    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654271    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654274    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654277    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 16:38:08.654275 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654280    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654282    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654285    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654288    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654291    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654294    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654296    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654299    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654301    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654304    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654306    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654309    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654312    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654314    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654317    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654320    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654322    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654325    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654327    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654330    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 16:38:08.654788 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654332    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654334    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654337    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654339    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654342    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654344    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654347    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654349    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654352    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654356    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654358    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654361    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654364    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654366    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654372    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654375    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654378    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654380    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654383    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654386    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 16:38:08.655276 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654389    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654391    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654394    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654397    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654399    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654402    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654404    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654407    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654411    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654416    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654420    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654424    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654427    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654430    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654432    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654435    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654438    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654441    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654443    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 16:38:08.655782 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654446    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654448    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654451    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654456    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654460    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654463    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654465    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654468    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654471    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654474    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654476    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654479    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654482    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654485    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654487    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654490    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654492    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654495    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654497    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654928    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 16:38:08.656236 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654934    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654937    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654940    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654943    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654946    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654948    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654951    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654954    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654956    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654959    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654963    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654967    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654970    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654972    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654975    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654980    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654984    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654987    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654990    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 16:38:08.656713 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654993    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654996    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.654998    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655001    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655004    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655006    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655009    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655013    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655015    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655018    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655021    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655023    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655026    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655028    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655031    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655034    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655036    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655039    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655042    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655044    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 16:38:08.657183 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655047    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655049    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655052    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655054    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655057    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655059    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655063    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655067    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655070    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655073    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655076    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655080    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655083    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655086    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655089    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655092    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655095    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655097    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655101    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 16:38:08.657697 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655104    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655106    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655109    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655111    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655114    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655117    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655119    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655122    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655124    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655126    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655129    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655131    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655134    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655137    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655139    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655142    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655145    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655147    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655150    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655153    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 16:38:08.658182 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655155    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655158    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655160    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655163    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655166    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655171    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655174    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655177    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655179    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655182    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655184    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655187    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655189    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655191    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655194    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655196    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655199    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655201    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655204    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655206    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 16:38:08.658650 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655288    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655301    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655311    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655318    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655324    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655330    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655335    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655339    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655342    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655345    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655349    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655352    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655355    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655357    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655361    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655363    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655369    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655372    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655376    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655379    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655382    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655385    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655388    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655392    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 19 16:38:08.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655394    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655397    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655400    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655403    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655406    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655409    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655412    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655415    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655418    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655421    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655424    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655427    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655430    2584 flags.go:64] FLAG: --enable-server="true"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655433    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655440    2584 flags.go:64] FLAG: --event-burst="100"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655443    2584 flags.go:64] FLAG: --event-qps="50"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655446    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655449    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655451    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655455    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655458    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655460    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655464    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655466    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655470    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 19 16:38:08.659896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655474    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655477    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655480    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655483    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655486    2584 flags.go:64] FLAG: --feature-gates=""
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655489    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655492    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655496    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655499    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655501    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655504    2584 flags.go:64] FLAG: --help="false"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655507    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655510    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655513    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655516    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655519    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655523    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655526    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655528    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655531    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655534    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655536    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655540    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655542    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 19 16:38:08.660509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655545    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655548    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655551    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655554    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655557    2584 flags.go:64] FLAG: --lock-file=""
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655559    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655562    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655566    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655572    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655576    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655579    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655581    2584 flags.go:64] FLAG: --logging-format="text"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655584    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655587    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655590    2584 flags.go:64] FLAG: --manifest-url=""
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655593    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655597    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655600    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655604    2584 flags.go:64] FLAG: --max-pods="110"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655606    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655609    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655612    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655615    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655618    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655621    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 19 16:38:08.661133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655623    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655631    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655633    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655637    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655640    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655643    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655648    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655651    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655654    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655657    2584 flags.go:64] FLAG: --port="10250"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655660    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655662    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a49ba3ebb2c696ce"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655665    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655668    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655672    2584 flags.go:64] FLAG: --register-node="true"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655675    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655679    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655682    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655685    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655688    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655691    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655694    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655697    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655700    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655703    2584 flags.go:64] FLAG: --runonce="false"
Jul 19 16:38:08.661732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655705    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655708    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655711    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655714    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655716    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655719    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655722    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655725    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655728    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655731    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655733    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655736    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655739    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655755    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655758    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655764    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655766    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655769    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655773    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655776    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655779    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655784    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655787    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655791    2584 flags.go:64] FLAG: --v="2"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655795    2584 flags.go:64] FLAG: --version="false"
Jul 19 16:38:08.662353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655800    2584 flags.go:64] FLAG: --vmodule=""
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655804    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.655807    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655938    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655942    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655945    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655948    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655951    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655953    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655956    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655959    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655961    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655964    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655966    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655969    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655971    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655974    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655976    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655979    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655982    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 16:38:08.663150 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655984    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655987    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655990    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655992    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655995    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.655999    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656002    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656005    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656009    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656012    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656017    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656019    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656022    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656026    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656028    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656031    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656034    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656037    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656039    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 16:38:08.664017 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656042    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656044    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656047    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656050    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656052    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656055    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656058    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656060    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656063    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656065    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656068    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656070    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656073    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656075    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656078    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656080    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656083    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656085    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656088    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656090    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 16:38:08.664833 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656093    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656097    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656100    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656104    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656106    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656109    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656112    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656115    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656118    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656122    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656125    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656128    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656131    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656134    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656136    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656139    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656142    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656144    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656147    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 16:38:08.665672 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656149    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656152    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656154    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656157    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656159    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656162    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656165    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656168    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656170    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656173    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656175    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656177    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656180    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656183    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656186    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656189    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656192    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656195    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656198    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656200    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 16:38:08.666495 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656203    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656206    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656209    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.656213    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.656918    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.665035    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.665055    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.665082    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.665091    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665175    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665187    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665192    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665198    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665203    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 16:38:08.667141 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665208    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665212    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665217    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665221    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665226    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665230    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665234    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665239    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665243    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665247    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665252    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665256    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665260    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665265    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665269    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665273    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665277    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665282    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665286    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 16:38:08.667707 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665290    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665294    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665298    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665303    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665307    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665314    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665318    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665323    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665327    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665332    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665336    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665341    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665345    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665349    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665353    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665357    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665362    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665366    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665370    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665374    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 16:38:08.668506 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665378    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665383    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665387    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665392    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665396    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665400    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665405    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665409    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665413    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665418    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665422    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665426    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665430    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665434    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665438    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665443    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665447    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665452    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665457    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 16:38:08.669100 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665461    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665466    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665470    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665474    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665479    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665486    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665491    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665496    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665501    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665506    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665510    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665515    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665519    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665523    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665527    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665531    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665536    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665540    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665544    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 16:38:08.669678 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665548    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665552    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665557    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665561    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665565    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665569    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665573    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665577    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665582    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665586    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665590    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665597    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665603    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665608    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665613    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665617    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 16:38:08.670171 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665621    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.665631    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665835    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665846    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665851    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665856    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665861    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665866    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665871    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665875    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665881    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665888    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665894    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665898    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 16:38:08.670590 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665903    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665907    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665911    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665918    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665923    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665928    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665933    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665938    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665943    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665947    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665952    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665956    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665961    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665966    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665971    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665975    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665980    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665985    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665989    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 16:38:08.670972 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665994    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.665998    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666002    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666006    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666010    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666014    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666018    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666023    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666027    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666032    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666036    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666040    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666044    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666048    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666052    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666056    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666060    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666068    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666073    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 16:38:08.671496 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666078    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666082    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666087    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666092    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666096    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666101    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666105    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666110    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666115    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666119    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666123    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666129    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666134    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666138    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666142    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666147    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666151    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666155    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666160    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666164    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 16:38:08.671982 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666168    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666172    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666176    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666180    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666184    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666189    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666193    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666197    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666202    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666206    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666210    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666214    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666218    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666222    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666226    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666230    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666235    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666239    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666244    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666248    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 16:38:08.672463 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666252    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666256    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666261    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666265    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666270    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666274    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666279    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666283    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:08.666287    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.666295    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.666462    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.669446    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.671207    2584 server.go:1019] "Starting client certificate rotation"
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.671316    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 19 16:38:08.673027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.671355    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 19 16:38:08.675146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.675127    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 19 16:38:08.676394 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.676374    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 19 16:38:08.688679 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.688657    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 19 16:38:08.691253 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.691224    2584 log.go:25] "Validated CRI v1 image API"
Jul 19 16:38:08.692100 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.692086    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 19 16:38:08.699144 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.699124    2584 fs.go:135] Filesystem UUIDs: map[02e59db4-d92c-4a27-a47b-0e151a711c6e:/dev/nvme0n1p3 603b1537-f9ba-4d31-bbc3-67be9a8a0a96:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 19 16:38:08.699212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.699144    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 19 16:38:08.704161 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.704141    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 19 16:38:08.704330 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.704207    2584 manager.go:216] Machine: {Timestamp:2026-07-19 16:38:08.702818663 +0000 UTC m=+0.383604859 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095168 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25141af3071ac8de174f07733575c4 SystemUUID:ec25141a-f307-1ac8-de17-4f07733575c4 BootID:c5d555dd-154c-482c-a903-def9e29df93a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:c8:0c:ad:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:c8:0c:ad:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:97:c6:16:27:63 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 19 16:38:08.704402 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.704331    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 19 16:38:08.704436 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.704415    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 19 16:38:08.707938 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.707909    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 19 16:38:08.708084 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.707941    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-78.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 19 16:38:08.708778 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.708766    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 19 16:38:08.708814 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.708779    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 19 16:38:08.708814 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.708788    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 19 16:38:08.708814 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.708803    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 19 16:38:08.710597 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.710586    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 19 16:38:08.715168 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715156    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 19 16:38:08.715294 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715284    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 19 16:38:08.715370 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715327    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 19 16:38:08.715370 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715351    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 19 16:38:08.715423 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715371    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 19 16:38:08.715423 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715386    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 19 16:38:08.715423 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715398    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 19 16:38:08.715783 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715772    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 19 16:38:08.715820 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.715795    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 19 16:38:08.718212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.718197    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 19 16:38:08.720701 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.720686    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 19 16:38:08.720768 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.720717    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 19 16:38:08.722610 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722599    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722616    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722622    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722628    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722637    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722643    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722651    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 19 16:38:08.722655 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722657    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 19 16:38:08.722850 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722663    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 19 16:38:08.722850 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722670    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 19 16:38:08.722850 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722678    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 19 16:38:08.722850 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.722683    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 19 16:38:08.723461 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.723452    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 19 16:38:08.723461 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.723461    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 19 16:38:08.724504 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.724484    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wk2qc"
Jul 19 16:38:08.726776 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.726732    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 19 16:38:08.726901 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.726887    2584 server.go:1268] "Started kubelet"
Jul 19 16:38:08.726965 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.726903    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-78.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 19 16:38:08.726965 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.726948    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 19 16:38:08.727054 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.726952    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 19 16:38:08.727054 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.727017    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 19 16:38:08.727155 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.726979    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 19 16:38:08.727712 ip-10-0-142-78 systemd[1]: Started Kubernetes Kubelet.
Jul 19 16:38:08.730306 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.730279    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 19 16:38:08.731125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.731107    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 19 16:38:08.733236 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.733215    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wk2qc"
Jul 19 16:38:08.733394 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.733378    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 19 16:38:08.736036 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.734937    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-78.ec2.internal.18c3be46d0a97016  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-78.ec2.internal,UID:ip-10-0-142-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-78.ec2.internal,},FirstTimestamp:2026-07-19 16:38:08.72660175 +0000 UTC m=+0.407387949,LastTimestamp:2026-07-19 16:38:08.72660175 +0000 UTC m=+0.407387949,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-78.ec2.internal,}"
Jul 19 16:38:08.741780 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.741761    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 19 16:38:08.741780 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.741771    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 19 16:38:08.741912 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.741858    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:08.742534 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742477    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 19 16:38:08.742534 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742481    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 19 16:38:08.742534 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742504    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 19 16:38:08.742716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742578    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 19 16:38:08.742716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742587    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 19 16:38:08.742716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742659    2584 factory.go:55] Registering systemd factory
Jul 19 16:38:08.742716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742672    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 19 16:38:08.742989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742974    2584 factory.go:153] Registering CRI-O factory
Jul 19 16:38:08.742989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.742989    2584 factory.go:223] Registration of the crio container factory successfully
Jul 19 16:38:08.743105 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.743045    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 19 16:38:08.743105 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.743072    2584 factory.go:103] Registering Raw factory
Jul 19 16:38:08.743105 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.743086    2584 manager.go:1196] Started watching for new ooms in manager
Jul 19 16:38:08.743944 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.743924    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 16:38:08.744473 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.744459    2584 manager.go:349] Starting recovery of all containers
Jul 19 16:38:08.746983 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.746958    2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-78.ec2.internal\" not found" node="ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.751130 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.751092    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 19 16:38:08.753268 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.753252    2584 manager.go:354] Recovery completed
Jul 19 16:38:08.757491 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.757476    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.760114 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760098    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.760169 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760129    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.760169 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760140    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.760716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760703    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 19 16:38:08.760716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760715    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 19 16:38:08.760837 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760729    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.760837 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.760732    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 19 16:38:08.761522 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.761508    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.761576 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.761533    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.761576 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.761543    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.762847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.762835    2584 policy_none.go:49] "None policy: Start"
Jul 19 16:38:08.762889 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.762854    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 19 16:38:08.762889 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.762864    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 19 16:38:08.764886 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.764866    2584 policy_none.go:47] "Start"
Jul 19 16:38:08.800657 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.798323    2584 manager.go:337] "Starting Device Plugin manager"
Jul 19 16:38:08.800657 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.798536    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 19 16:38:08.800657 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.798551    2584 server.go:87] "Starting device plugin registration server"
Jul 19 16:38:08.801246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.800813    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 19 16:38:08.801246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.800824    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 19 16:38:08.801246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.800940    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 19 16:38:08.801246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.801022    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 19 16:38:08.801246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.801033    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 19 16:38:08.801618 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.801596    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 19 16:38:08.801727 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.801649    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:08.850456 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.850428    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 19 16:38:08.852633 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.850468    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 19 16:38:08.852633 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.850488    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 19 16:38:08.852633 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.850519    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 19 16:38:08.852633 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.852554    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 16:38:08.901870 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.901785    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.902793 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.902757    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.902793 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.902788    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.902793 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.902798    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.902963 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.902821    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.908696 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.908680    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.908784 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.908701    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-78.ec2.internal\": node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:08.920726 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.920706    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:08.951592 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.951558    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"]
Jul 19 16:38:08.951692 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.951645    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.953719 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.953703    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.953828 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.953731    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.953828 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.953765    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.955952 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.955940    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.956050 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.956038    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.956085 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.956065    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.957337 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.957321    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.957409 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.957348    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.957409 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.957358    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.957409 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.957321    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.957506 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.957430    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.957506 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.957451    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.959555 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.959540    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.959683 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.959566    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 16:38:08.960246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.960218    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 16:38:08.960330 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.960250    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 16:38:08.960330 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:08.960260    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeHasSufficientPID"
Jul 19 16:38:08.989542 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.989521    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-78.ec2.internal\" not found" node="ip-10-0-142-78.ec2.internal"
Jul 19 16:38:08.993004 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:08.992988    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-78.ec2.internal\" not found" node="ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.020816 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.020790    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.043613 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.043590    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8bc2a20e77f7595ccd462b1cae835411-config\") pod \"kube-apiserver-proxy-ip-10-0-142-78.ec2.internal\" (UID: \"8bc2a20e77f7595ccd462b1cae835411\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.043774 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.043617    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ae521fb49591405044742393ea51e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"5ae521fb49591405044742393ea51e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.043774 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.043637    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ae521fb49591405044742393ea51e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"5ae521fb49591405044742393ea51e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.121130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.121098    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.144491 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.144453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ae521fb49591405044742393ea51e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"5ae521fb49591405044742393ea51e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.144491 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.144489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8bc2a20e77f7595ccd462b1cae835411-config\") pod \"kube-apiserver-proxy-ip-10-0-142-78.ec2.internal\" (UID: \"8bc2a20e77f7595ccd462b1cae835411\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.144653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.144506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ae521fb49591405044742393ea51e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"5ae521fb49591405044742393ea51e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.144653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.144563    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ae521fb49591405044742393ea51e08-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"5ae521fb49591405044742393ea51e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.144653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.144612    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8bc2a20e77f7595ccd462b1cae835411-config\") pod \"kube-apiserver-proxy-ip-10-0-142-78.ec2.internal\" (UID: \"8bc2a20e77f7595ccd462b1cae835411\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.144653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.144644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5ae521fb49591405044742393ea51e08-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal\" (UID: \"5ae521fb49591405044742393ea51e08\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.221930 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.221859    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.292373 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.292337    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.295504 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.295487    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:09.322817 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.322788    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.423320 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.423274    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.523933 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.523860    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.624356 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.624325    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.669868 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.669840    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 19 16:38:09.670301 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.670004    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 19 16:38:09.670301 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.670032    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 19 16:38:09.725326 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.725301    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.740462 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.740420    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 16:33:10 +0000 UTC" deadline="2028-02-24 21:43:59.225536642 +0000 UTC"
Jul 19 16:38:09.740462 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.740458    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14045h5m49.485082045s"
Jul 19 16:38:09.826373 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.826303    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:09.846432 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:09.846396    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc2a20e77f7595ccd462b1cae835411.slice/crio-242b2209d6b9e9d3689729f8757bdfe89fca83793026f8871857f3230783a7b6 WatchSource:0}: Error finding container 242b2209d6b9e9d3689729f8757bdfe89fca83793026f8871857f3230783a7b6: Status 404 returned error can't find the container with id 242b2209d6b9e9d3689729f8757bdfe89fca83793026f8871857f3230783a7b6
Jul 19 16:38:09.847321 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:09.847291    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae521fb49591405044742393ea51e08.slice/crio-ee2cf8f545d7f682bbc835c905ce90c86ccb92ae5e8da3b023672ca70af3c639 WatchSource:0}: Error finding container ee2cf8f545d7f682bbc835c905ce90c86ccb92ae5e8da3b023672ca70af3c639: Status 404 returned error can't find the container with id ee2cf8f545d7f682bbc835c905ce90c86ccb92ae5e8da3b023672ca70af3c639
Jul 19 16:38:09.852982 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.852966    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 16:38:09.853683 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.853569    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" event={"ID":"5ae521fb49591405044742393ea51e08","Type":"ContainerStarted","Data":"ee2cf8f545d7f682bbc835c905ce90c86ccb92ae5e8da3b023672ca70af3c639"}
Jul 19 16:38:09.854563 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:09.854542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal" event={"ID":"8bc2a20e77f7595ccd462b1cae835411","Type":"ContainerStarted","Data":"242b2209d6b9e9d3689729f8757bdfe89fca83793026f8871857f3230783a7b6"}
Jul 19 16:38:09.927331 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:09.927293    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:10.027909 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:10.027866    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:10.128406 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:10.128331    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-78.ec2.internal\" not found"
Jul 19 16:38:11.958060 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:11.958030    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 16:38:11.984827 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:11.984806    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 16:38:12.018000 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.017972    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:12.027630 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.027608    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 19 16:38:12.028746 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.028731    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal"
Jul 19 16:38:12.037772 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.037753    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 19 16:38:12.492547 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.492516    2584 apiserver.go:52] "Watching apiserver"
Jul 19 16:38:12.503499 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.503467    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 19 16:38:12.505190 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.505154    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-24cj4","kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr","openshift-cluster-node-tuning-operator/tuned-n9gfs","openshift-dns/node-resolver-2g5qt","openshift-multus/multus-additional-cni-plugins-xrzwf","openshift-multus/network-metrics-daemon-xqq8r","kube-system/global-pull-secret-syncer-pmwch","openshift-image-registry/node-ca-qbv9q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal","openshift-multus/multus-mbbtj","openshift-network-diagnostics/network-check-target-46r78","openshift-network-operator/iptables-alerter-j9f75","openshift-ovn-kubernetes/ovnkube-node-b4lp2"]
Jul 19 16:38:12.508156 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.507753    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:12.508156 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.507837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:12.510640 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.510047    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.513755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.513309    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.513755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.513555    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.513755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.513583    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mq5c9\""
Jul 19 16:38:12.513755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.513597    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 19 16:38:12.513755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.513597    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.514809 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.514475    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.515936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.515748    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 19 16:38:12.515936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.515910    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-79v6x\""
Jul 19 16:38:12.516142 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.515960    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.516204 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.516192    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.516621 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.516602    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.516786 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.516768    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7cz4s\""
Jul 19 16:38:12.516836 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.516812    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.517010 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.516995    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.517892 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.517876    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 19 16:38:12.519314 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.519269    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.519410 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.519341    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.519610 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.519592    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.519610 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.519605    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4r9c4\""
Jul 19 16:38:12.521809 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.521793    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 19 16:38:12.522114 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.522091    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.522205 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.522183    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 19 16:38:12.522265 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.522227    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.522265 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.522241    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wbnzx\""
Jul 19 16:38:12.522475 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.522455    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.522571 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.522497    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 19 16:38:12.525325 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.525301    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 19 16:38:12.525429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.525340    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.525525 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.525508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.525581 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.525569    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.526011 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.525993    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xmqbx\""
Jul 19 16:38:12.527899 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.527877    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 19 16:38:12.528389 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7hpm8\""
Jul 19 16:38:12.528508 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528483    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:12.528508 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528502    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.528627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1072ee5d-843c-4c96-967f-240ac7e610ef-iptables-alerter-script\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.528627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-socket-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.528627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528490    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 19 16:38:12.528627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528591    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysctl-conf\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.528627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528615    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-host\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528638    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528662    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528688    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72szl\" (UniqueName: \"kubernetes.io/projected/1072ee5d-843c-4c96-967f-240ac7e610ef-kube-api-access-72szl\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-device-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528759    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-modprobe-d\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528783    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-systemd\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-lib-modules\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqsf\" (UniqueName: \"kubernetes.io/projected/df49f583-c7e0-4367-af64-995e7bccf5ec-kube-api-access-rpqsf\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1072ee5d-843c-4c96-967f-240ac7e610ef-host-slash\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-sys-fs\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.528890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-kubernetes\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysctl-d\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528955    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df49f583-c7e0-4367-af64-995e7bccf5ec-hosts-file\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.528999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529024    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysconfig\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529081    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-tuned\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbm7c\" (UniqueName: \"kubernetes.io/projected/79070897-558c-4406-a143-39cf00b7a155-kube-api-access-vbm7c\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529174    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-run\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529217    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.529393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529248    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2xd\" (UniqueName: \"kubernetes.io/projected/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-kube-api-access-5p2xd\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-registration-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529537    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwqn\" (UniqueName: \"kubernetes.io/projected/1d61ddf7-7492-42b9-9c8b-4f69c4838231-kube-api-access-rqwqn\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529573    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-cni-binary-copy\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529600    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a4ba7cb-3517-4dd2-9442-8b050f343616-tmp\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df49f583-c7e0-4367-af64-995e7bccf5ec-tmp-dir\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-system-cni-dir\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.529748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529718    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-cnibin\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.530050 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-sys\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.530050 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529797    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-var-lib-kubelet\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.530050 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/2a4ba7cb-3517-4dd2-9442-8b050f343616-kube-api-access-zx8h7\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.530050 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.529857    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-os-release\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531299    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531334    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531333    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531570    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9td7v\""
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531672    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.531723    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.532291 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.531788    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:12.532726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.532438    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 19 16:38:12.532726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.532509    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 19 16:38:12.532862 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.532842    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.535135 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.535114    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8lv9m\""
Jul 19 16:38:12.535302 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.535266    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 19 16:38:12.535687 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.535657    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:12.535779 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.535721    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:12.549743 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.549628    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2xtzn"
Jul 19 16:38:12.557337 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.557246    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2xtzn"
Jul 19 16:38:12.619451 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.619421    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 19 16:38:12.630056 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630019    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbm7c\" (UniqueName: \"kubernetes.io/projected/79070897-558c-4406-a143-39cf00b7a155-kube-api-access-vbm7c\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.630056 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-node-log\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.630353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldb8\" (UniqueName: \"kubernetes.io/projected/1b270190-1d31-44c3-9189-ee59c30ad802-kube-api-access-jldb8\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.630353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-cni-bin\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.630353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630267    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-cni-multus\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.630353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630329    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-conf-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.630353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630359    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-run\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.630622 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630421    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.630622 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-slash\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.630622 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630582    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-run\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.630773 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-cni-bin\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.630773 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630658    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-cni-netd\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.630773 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-hostroot\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.630773 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630708    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47cb1017-bf16-49de-acdc-4f766a1845cc-konnectivity-ca\") pod \"konnectivity-agent-24cj4\" (UID: \"47cb1017-bf16-49de-acdc-4f766a1845cc\") " pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.630773 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630737    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwqn\" (UniqueName: \"kubernetes.io/projected/1d61ddf7-7492-42b9-9c8b-4f69c4838231-kube-api-access-rqwqn\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630779    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-var-lib-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630812    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630840    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m72w\" (UniqueName: \"kubernetes.io/projected/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-kube-api-access-8m72w\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630868    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df49f583-c7e0-4367-af64-995e7bccf5ec-tmp-dir\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-system-cni-dir\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.630942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-cnibin\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.631053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631000    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-cnibin\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-log-socket\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-system-cni-dir\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631117    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47cb1017-bf16-49de-acdc-4f766a1845cc-agent-certs\") pod \"konnectivity-agent-24cj4\" (UID: \"47cb1017-bf16-49de-acdc-4f766a1845cc\") " pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-var-lib-kubelet\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/2a4ba7cb-3517-4dd2-9442-8b050f343616-kube-api-access-zx8h7\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631192    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df49f583-c7e0-4367-af64-995e7bccf5ec-tmp-dir\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631236    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-var-lib-kubelet\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631268    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-kubelet-config\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-cni-binary-copy\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-serviceca\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-kubelet\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631419    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-host\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.631477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-device-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-modprobe-d\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631567    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-systemd\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631594    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-kubelet\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1072ee5d-843c-4c96-967f-240ac7e610ef-host-slash\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631670    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-sys-fs\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631694    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysctl-d\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df49f583-c7e0-4367-af64-995e7bccf5ec-hosts-file\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631743    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-modprobe-d\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-socket-dir-parent\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-device-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631827    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1072ee5d-843c-4c96-967f-240ac7e610ef-host-slash\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-host\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-sys-fs\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-systemd\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysconfig\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631978    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-tuned\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df49f583-c7e0-4367-af64-995e7bccf5ec-hosts-file\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-etc-kubernetes\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632019    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysconfig\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-os-release\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.631992    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysctl-d\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632052    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2xd\" (UniqueName: \"kubernetes.io/projected/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-kube-api-access-5p2xd\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-registration-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632115    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-cni-binary-copy\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632137    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-run-netns\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-etc-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632212    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-dbus\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632236    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.632998 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632263    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-netns\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a4ba7cb-3517-4dd2-9442-8b050f343616-tmp\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632339    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632376    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-env-overrides\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632400    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-daemon-config\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hldc\" (UniqueName: \"kubernetes.io/projected/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-kube-api-access-2hldc\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-sys\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-os-release\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632517    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-systemd\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632539    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b270190-1d31-44c3-9189-ee59c30ad802-ovn-node-metrics-cert\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632562    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-systemd-units\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-ovn\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632610    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632666    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1072ee5d-843c-4c96-967f-240ac7e610ef-iptables-alerter-script\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632678    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79070897-558c-4406-a143-39cf00b7a155-cni-binary-copy\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.633723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-socket-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysctl-conf\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72szl\" (UniqueName: \"kubernetes.io/projected/1072ee5d-843c-4c96-967f-240ac7e610ef-kube-api-access-72szl\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-lib-modules\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632794    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-socket-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632804    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqsf\" (UniqueName: \"kubernetes.io/projected/df49f583-c7e0-4367-af64-995e7bccf5ec-kube-api-access-rpqsf\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632829    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-ovnkube-config\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632880    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632885    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-ovnkube-script-lib\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632313    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d61ddf7-7492-42b9-9c8b-4f69c4838231-registration-dir\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632909    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-cnibin\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632934    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-k8s-cni-cncf-io\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632952    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-kubernetes\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632969    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-system-cni-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.632988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-cni-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633018    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-multus-certs\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.634477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-host\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-sysctl-conf\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633448    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-sys\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-lib-modules\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633517    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-os-release\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633564    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-kubernetes\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.633682    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.633784    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:38:13.133742571 +0000 UTC m=+3.038993995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.633800    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79070897-558c-4406-a143-39cf00b7a155-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.635259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.634066    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1072ee5d-843c-4c96-967f-240ac7e610ef-iptables-alerter-script\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.637000 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.636976    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a4ba7cb-3517-4dd2-9442-8b050f343616-tmp\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.638318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.638295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a4ba7cb-3517-4dd2-9442-8b050f343616-etc-tuned\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.638492 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.638469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbm7c\" (UniqueName: \"kubernetes.io/projected/79070897-558c-4406-a143-39cf00b7a155-kube-api-access-vbm7c\") pod \"multus-additional-cni-plugins-xrzwf\" (UID: \"79070897-558c-4406-a143-39cf00b7a155\") " pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.639750 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.639731    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2xd\" (UniqueName: \"kubernetes.io/projected/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-kube-api-access-5p2xd\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:12.639847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.639809    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwqn\" (UniqueName: \"kubernetes.io/projected/1d61ddf7-7492-42b9-9c8b-4f69c4838231-kube-api-access-rqwqn\") pod \"aws-ebs-csi-driver-node-dfcnr\" (UID: \"1d61ddf7-7492-42b9-9c8b-4f69c4838231\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.640544 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.640525    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72szl\" (UniqueName: \"kubernetes.io/projected/1072ee5d-843c-4c96-967f-240ac7e610ef-kube-api-access-72szl\") pod \"iptables-alerter-j9f75\" (UID: \"1072ee5d-843c-4c96-967f-240ac7e610ef\") " pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.643583 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.643552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/2a4ba7cb-3517-4dd2-9442-8b050f343616-kube-api-access-zx8h7\") pod \"tuned-n9gfs\" (UID: \"2a4ba7cb-3517-4dd2-9442-8b050f343616\") " pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.643879 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.643860    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqsf\" (UniqueName: \"kubernetes.io/projected/df49f583-c7e0-4367-af64-995e7bccf5ec-kube-api-access-rpqsf\") pod \"node-resolver-2g5qt\" (UID: \"df49f583-c7e0-4367-af64-995e7bccf5ec\") " pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.733975 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.733938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-systemd-units\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.733975 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.733974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-ovn\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734001    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-ovnkube-config\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-ovnkube-script-lib\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-systemd-units\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-ovn\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-run-ovn-kubernetes\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734130    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-cnibin\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-k8s-cni-cncf-io\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-system-cni-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734198    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-cni-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734212    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-cnibin\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-multus-certs\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-k8s-cni-cncf-io\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734251    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-host\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-system-cni-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-node-log\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-multus-certs\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-host\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734331    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-node-log\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldb8\" (UniqueName: \"kubernetes.io/projected/1b270190-1d31-44c3-9189-ee59c30ad802-kube-api-access-jldb8\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734389    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-cni-bin\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-cni-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-cni-multus\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734446    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-cni-bin\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734445    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-conf-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-conf-dir\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-slash\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-slash\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.734656 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734508    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-cni-bin\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-cni-multus\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734536    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-cni-bin\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734541    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-cni-netd\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734574    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-cni-netd\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734600    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-hostroot\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47cb1017-bf16-49de-acdc-4f766a1845cc-konnectivity-ca\") pod \"konnectivity-agent-24cj4\" (UID: \"47cb1017-bf16-49de-acdc-4f766a1845cc\") " pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-hostroot\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734690    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-var-lib-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734722    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734726    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-ovnkube-script-lib\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734727    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-ovnkube-config\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-var-lib-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m72w\" (UniqueName: \"kubernetes.io/projected/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-kube-api-access-8m72w\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734890    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-log-socket\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734921    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-log-socket\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.734939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47cb1017-bf16-49de-acdc-4f766a1845cc-agent-certs\") pod \"konnectivity-agent-24cj4\" (UID: \"47cb1017-bf16-49de-acdc-4f766a1845cc\") " pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.735416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735091    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-kubelet-config\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735121    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-cni-binary-copy\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-serviceca\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-kubelet\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-kubelet\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735259    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-kubelet\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-socket-dir-parent\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735380    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-socket-dir-parent\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735202    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/47cb1017-bf16-49de-acdc-4f766a1845cc-konnectivity-ca\") pod \"konnectivity-agent-24cj4\" (UID: \"47cb1017-bf16-49de-acdc-4f766a1845cc\") " pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735484    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-var-lib-kubelet\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-kubelet-config\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-etc-kubernetes\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735688    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-cni-binary-copy\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735703    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-os-release\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-run-netns\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735736    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-etc-kubernetes\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-os-release\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735770    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-etc-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735799    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-dbus\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735801    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-host-run-netns\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735807    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-etc-openvswitch\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735848    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-netns\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735871    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-env-overrides\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735877    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-dbus\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-daemon-config\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-host-run-netns\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735933    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hldc\" (UniqueName: \"kubernetes.io/projected/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-kube-api-access-2hldc\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.735959    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735998    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-systemd\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.736028    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:38:13.236010013 +0000 UTC m=+3.141261438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:12.736920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.735963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b270190-1d31-44c3-9189-ee59c30ad802-run-systemd\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.737680 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.736061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b270190-1d31-44c3-9189-ee59c30ad802-ovn-node-metrics-cert\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.737680 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.736357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b270190-1d31-44c3-9189-ee59c30ad802-env-overrides\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.737680 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.736437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-multus-daemon-config\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.737680 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.736439    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-serviceca\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.738071 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.738051    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/47cb1017-bf16-49de-acdc-4f766a1845cc-agent-certs\") pod \"konnectivity-agent-24cj4\" (UID: \"47cb1017-bf16-49de-acdc-4f766a1845cc\") " pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.738898 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.738876    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b270190-1d31-44c3-9189-ee59c30ad802-ovn-node-metrics-cert\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.740780 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.740757    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:12.740780 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.740780    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:12.740950 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.740793    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:12.740950 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:12.740850    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:38:13.240832205 +0000 UTC m=+3.146083613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:12.743303 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.743213    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m72w\" (UniqueName: \"kubernetes.io/projected/c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d-kube-api-access-8m72w\") pod \"multus-mbbtj\" (UID: \"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d\") " pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:12.743498 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.743481    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldb8\" (UniqueName: \"kubernetes.io/projected/1b270190-1d31-44c3-9189-ee59c30ad802-kube-api-access-jldb8\") pod \"ovnkube-node-b4lp2\" (UID: \"1b270190-1d31-44c3-9189-ee59c30ad802\") " pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.744404 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.744381    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hldc\" (UniqueName: \"kubernetes.io/projected/900bb980-a550-45b4-a0ee-06d3e4c3f2d4-kube-api-access-2hldc\") pod \"node-ca-qbv9q\" (UID: \"900bb980-a550-45b4-a0ee-06d3e4c3f2d4\") " pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.752930 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.752906    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 16:38:12.826116 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.826085    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j9f75"
Jul 19 16:38:12.834944 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.834921    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr"
Jul 19 16:38:12.848660 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.848629    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9gfs"
Jul 19 16:38:12.856491 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.856463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2g5qt"
Jul 19 16:38:12.857153 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.857134    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 16:38:12.864579 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.864559    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xrzwf"
Jul 19 16:38:12.872345 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.872325    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qbv9q"
Jul 19 16:38:12.880066 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.880044    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:12.888014 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.887993    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:12.894527 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:12.894507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mbbtj"
Jul 19 16:38:13.140611 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.140535    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:13.140781 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.140659    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:13.140781 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.140724    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:38:14.140706946 +0000 UTC m=+4.045958348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:13.241012 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.240976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:13.241012 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.241016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:13.241250 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.241132    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:13.241250 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.241144    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:13.241250 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.241167    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:13.241250 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.241177    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:13.241250 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.241182    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:38:14.24116998 +0000 UTC m=+4.146421382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:13.241250 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:13.241224    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:38:14.241208156 +0000 UTC m=+4.146459558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:13.561220 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.561117    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 16:33:12 +0000 UTC" deadline="2028-02-20 05:21:54.096610639 +0000 UTC"
Jul 19 16:38:13.561220 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.561170    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13932h43m40.53544413s"
Jul 19 16:38:13.602491 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.602458    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cb1017_bf16_49de_acdc_4f766a1845cc.slice/crio-699fd829374714e93cf91453175483a6bdd6048fe95dc51b68d2ceb77f4f4726 WatchSource:0}: Error finding container 699fd829374714e93cf91453175483a6bdd6048fe95dc51b68d2ceb77f4f4726: Status 404 returned error can't find the container with id 699fd829374714e93cf91453175483a6bdd6048fe95dc51b68d2ceb77f4f4726
Jul 19 16:38:13.603738 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.603697    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79070897_558c_4406_a143_39cf00b7a155.slice/crio-92202f29000f2e822e6f4dec1fb521cc867d53714d194ad9d2ea3065df885fdd WatchSource:0}: Error finding container 92202f29000f2e822e6f4dec1fb521cc867d53714d194ad9d2ea3065df885fdd: Status 404 returned error can't find the container with id 92202f29000f2e822e6f4dec1fb521cc867d53714d194ad9d2ea3065df885fdd
Jul 19 16:38:13.604911 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.604890    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4ba7cb_3517_4dd2_9442_8b050f343616.slice/crio-93718307fb6d9c7bbcb13c807fbd0705989299e0a9f117666c291253282581a6 WatchSource:0}: Error finding container 93718307fb6d9c7bbcb13c807fbd0705989299e0a9f117666c291253282581a6: Status 404 returned error can't find the container with id 93718307fb6d9c7bbcb13c807fbd0705989299e0a9f117666c291253282581a6
Jul 19 16:38:13.607433 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.607413    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072ee5d_843c_4c96_967f_240ac7e610ef.slice/crio-581a8142fc9707a1e07a5581f42732e8ffb684d143142be9e69ab7150ed48c9c WatchSource:0}: Error finding container 581a8142fc9707a1e07a5581f42732e8ffb684d143142be9e69ab7150ed48c9c: Status 404 returned error can't find the container with id 581a8142fc9707a1e07a5581f42732e8ffb684d143142be9e69ab7150ed48c9c
Jul 19 16:38:13.608156 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.608129    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b270190_1d31_44c3_9189_ee59c30ad802.slice/crio-a2e586dfcb60b38c3455301a4fd3b1e9811115e62132cba86467cd61f9786ff9 WatchSource:0}: Error finding container a2e586dfcb60b38c3455301a4fd3b1e9811115e62132cba86467cd61f9786ff9: Status 404 returned error can't find the container with id a2e586dfcb60b38c3455301a4fd3b1e9811115e62132cba86467cd61f9786ff9
Jul 19 16:38:13.610050 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.610030    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900bb980_a550_45b4_a0ee_06d3e4c3f2d4.slice/crio-cfcf9db3bc7cb347242680eb4965cb52b6c2198d17d8b2f092a1e8bfc95b8d0b WatchSource:0}: Error finding container cfcf9db3bc7cb347242680eb4965cb52b6c2198d17d8b2f092a1e8bfc95b8d0b: Status 404 returned error can't find the container with id cfcf9db3bc7cb347242680eb4965cb52b6c2198d17d8b2f092a1e8bfc95b8d0b
Jul 19 16:38:13.611501 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.611477    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d61ddf7_7492_42b9_9c8b_4f69c4838231.slice/crio-299cae7db372109906a0e854340c88bf7f832a8f4f5813a705157476f6fd3514 WatchSource:0}: Error finding container 299cae7db372109906a0e854340c88bf7f832a8f4f5813a705157476f6fd3514: Status 404 returned error can't find the container with id 299cae7db372109906a0e854340c88bf7f832a8f4f5813a705157476f6fd3514
Jul 19 16:38:13.612756 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:13.612737    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc013d9b4_d2e8_4ff6_9d38_bf57aa458a6d.slice/crio-38a04b0bb3c34a61abf110dd2ff43bed2b88d9d97a6e8d51695c73cd835dc2a7 WatchSource:0}: Error finding container 38a04b0bb3c34a61abf110dd2ff43bed2b88d9d97a6e8d51695c73cd835dc2a7: Status 404 returned error can't find the container with id 38a04b0bb3c34a61abf110dd2ff43bed2b88d9d97a6e8d51695c73cd835dc2a7
Jul 19 16:38:13.633747 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.633721    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerStarted","Data":"92202f29000f2e822e6f4dec1fb521cc867d53714d194ad9d2ea3065df885fdd"}
Jul 19 16:38:13.634862 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.634843    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24cj4" event={"ID":"47cb1017-bf16-49de-acdc-4f766a1845cc","Type":"ContainerStarted","Data":"699fd829374714e93cf91453175483a6bdd6048fe95dc51b68d2ceb77f4f4726"}
Jul 19 16:38:13.635972 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.635948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mbbtj" event={"ID":"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d","Type":"ContainerStarted","Data":"38a04b0bb3c34a61abf110dd2ff43bed2b88d9d97a6e8d51695c73cd835dc2a7"}
Jul 19 16:38:13.637160 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.637135    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"a2e586dfcb60b38c3455301a4fd3b1e9811115e62132cba86467cd61f9786ff9"}
Jul 19 16:38:13.638698 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.638674    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qbv9q" event={"ID":"900bb980-a550-45b4-a0ee-06d3e4c3f2d4","Type":"ContainerStarted","Data":"cfcf9db3bc7cb347242680eb4965cb52b6c2198d17d8b2f092a1e8bfc95b8d0b"}
Jul 19 16:38:13.640255 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.640230    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j9f75" event={"ID":"1072ee5d-843c-4c96-967f-240ac7e610ef","Type":"ContainerStarted","Data":"581a8142fc9707a1e07a5581f42732e8ffb684d143142be9e69ab7150ed48c9c"}
Jul 19 16:38:13.641452 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.641417    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9gfs" event={"ID":"2a4ba7cb-3517-4dd2-9442-8b050f343616","Type":"ContainerStarted","Data":"93718307fb6d9c7bbcb13c807fbd0705989299e0a9f117666c291253282581a6"}
Jul 19 16:38:13.643588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.643563    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2g5qt" event={"ID":"df49f583-c7e0-4367-af64-995e7bccf5ec","Type":"ContainerStarted","Data":"9b4552d9a49cdb54af235a3f18397c8cca1609a0ee72c8bb1b915434f3c94ac0"}
Jul 19 16:38:13.645238 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:13.645219    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr" event={"ID":"1d61ddf7-7492-42b9-9c8b-4f69c4838231","Type":"ContainerStarted","Data":"299cae7db372109906a0e854340c88bf7f832a8f4f5813a705157476f6fd3514"}
Jul 19 16:38:14.149956 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.149039    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:14.149956 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.149431    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:14.149956 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.149495    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:38:16.149473842 +0000 UTC m=+6.054725247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:14.249685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.249618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:14.249685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.249677    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:14.249907 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.249820    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:14.249907 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.249879    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:38:16.249861612 +0000 UTC m=+6.155113028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:14.250383 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.250363    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:14.250383 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.250387    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:14.250565 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.250400    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:14.250565 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.250448    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:38:16.250432725 +0000 UTC m=+6.155684129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:14.561757 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.561632    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 16:33:12 +0000 UTC" deadline="2027-12-17 18:36:35.398444636 +0000 UTC"
Jul 19 16:38:14.561757 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.561694    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12385h58m20.836755559s"
Jul 19 16:38:14.629393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.629362    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:14.629561 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.629490    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:14.629867 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.629844    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:14.629999 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.629979    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:14.630090 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.630076    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:14.630183 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:14.630166    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:14.681306 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.681009    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ae521fb49591405044742393ea51e08" containerID="4d5e467e593729d28514d638203d6f41a3529e33b15495c873626f90b93ad362" exitCode=0
Jul 19 16:38:14.681306 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.681113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" event={"ID":"5ae521fb49591405044742393ea51e08","Type":"ContainerDied","Data":"4d5e467e593729d28514d638203d6f41a3529e33b15495c873626f90b93ad362"}
Jul 19 16:38:14.698269 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:14.698235    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal" event={"ID":"8bc2a20e77f7595ccd462b1cae835411","Type":"ContainerStarted","Data":"acc8534dec118fe72419866cbfb09e1a082db30a0a4c896d591095f7207f5ac8"}
Jul 19 16:38:15.720385 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:15.720158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" event={"ID":"5ae521fb49591405044742393ea51e08","Type":"ContainerStarted","Data":"9cd691221d9da2af9b9075d927bd008b3e570d5ccf01dea6e82dc4a2ba77bcfb"}
Jul 19 16:38:15.735751 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:15.734450    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-78.ec2.internal" podStartSLOduration=3.734426306 podStartE2EDuration="3.734426306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:14.711692163 +0000 UTC m=+4.616943590" watchObservedRunningTime="2026-07-19 16:38:15.734426306 +0000 UTC m=+5.639677730"
Jul 19 16:38:15.735751 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:15.735067    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-78.ec2.internal" podStartSLOduration=3.735059016 podStartE2EDuration="3.735059016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:15.734221625 +0000 UTC m=+5.639473053" watchObservedRunningTime="2026-07-19 16:38:15.735059016 +0000 UTC m=+5.640310443"
Jul 19 16:38:16.167360 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:16.167006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:16.167360 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.167205    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:16.167360 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.167268    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:38:20.167250009 +0000 UTC m=+10.072501414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:16.268269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:16.268354    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.268517    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.268582    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:38:20.268563262 +0000 UTC m=+10.173814670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.268991    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.269009    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.269022    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:16.269130 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.269069    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:38:20.269054251 +0000 UTC m=+10.174305657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:16.628459 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:16.628381    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:16.628622 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:16.628391    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:16.628622 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.628514    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:16.628622 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:16.628525    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:16.628622 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.628611    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:16.628835 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:16.628701    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:18.626765 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:18.626729    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:18.627272 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:18.626859    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:18.627272 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:18.626941    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:18.627272 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:18.627047    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:18.627272 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:18.626729    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:18.627272 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:18.627166    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:20.204567 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:20.204530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:20.205053 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.204702    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:20.205053 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.204778    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:38:28.20475994 +0000 UTC m=+18.110011341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:20.305251 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:20.305217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:20.305451 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:20.305269    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:20.305451 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.305413    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:20.305451 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.305423    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:20.305451 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.305452    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:20.305642 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.305466    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:20.305642 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.305475    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:38:28.305455631 +0000 UTC m=+18.210707056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:20.305642 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.305516    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:38:28.305500105 +0000 UTC m=+18.210751511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:20.628646 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:20.627850    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:20.628646 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.627975    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:20.628646 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:20.628361    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:20.628646 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.628457    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:20.628646 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:20.628500    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:20.628646 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:20.628564    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:22.626935 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:22.626297    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:22.626935 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:22.626433    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:22.626935 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:22.626816    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:22.626935 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:22.626887    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:22.627575 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:22.627025    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:22.627575 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:22.627123    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:24.626620 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:24.626572    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:24.626620 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:24.626580    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:24.627118 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:24.626659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:24.627118 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:24.626796    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:24.627118 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:24.626910    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:24.627118 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:24.626981    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:26.626329 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:26.626295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:26.626776 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:26.626351    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:26.626776 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:26.626297    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:26.626776 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:26.626444    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:26.626776 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:26.626534    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:26.630218 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:26.627522    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:28.262819 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:28.262713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:28.263241 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.262887    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:28.263241 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.262953    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:38:44.262935841 +0000 UTC m=+34.168187243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:28.363793 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:28.363754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:28.364035 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:28.363849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:28.364035 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.363933    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:28.364035 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.363978    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:28.364035 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.363998    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:28.364035 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.364009    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:28.364240 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.364011    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:38:44.363989976 +0000 UTC m=+34.269241390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:28.364240 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.364067    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:38:44.364050901 +0000 UTC m=+34.269302303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:28.627295 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:28.627192    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:28.627295 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:28.627247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:28.627295 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:28.627254    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:28.627551 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.627381    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:28.627551 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.627462    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:28.627651 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:28.627567    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:30.628185 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:30.628135    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:30.628646 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:30.628235    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:30.628646 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:30.628328    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:30.628646 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:30.628342    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:30.628646 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:30.628371    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:30.628646 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:30.628437    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:32.626304 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:32.626262    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:32.626673 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:32.626266    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:32.626673 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:32.626400    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:32.626673 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:32.626268    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:32.626673 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:32.626519    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:32.626673 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:32.626611    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:32.762436 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:32.762345    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr" event={"ID":"1d61ddf7-7492-42b9-9c8b-4f69c4838231","Type":"ContainerStarted","Data":"94841b1f2b5f948d2961c7d246ab0835dbfa326b8a2352c2bdd9bd3ef038189b"}
Jul 19 16:38:32.764505 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:32.763973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-24cj4" event={"ID":"47cb1017-bf16-49de-acdc-4f766a1845cc","Type":"ContainerStarted","Data":"0dcb892dec87a2816884ccad291f3bd6fa837c8eef0fce85457142fd571856fa"}
Jul 19 16:38:32.778940 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:32.778533    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-24cj4" podStartSLOduration=8.546104396 podStartE2EDuration="22.778514471s" totalImagesPullingTime="14.232410075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:32.778231908 +0000 UTC m=+22.683483336" watchObservedRunningTime="2026-07-19 16:38:32.778514471 +0000 UTC m=+22.683765897"
Jul 19 16:38:33.767321 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.767041    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mbbtj" event={"ID":"c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d","Type":"ContainerStarted","Data":"11fb23a1eec6f1c9ba800298b6ebcabddb61e95215893cbf1ad357e31c8418c6"}
Jul 19 16:38:33.769706 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.769688    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:38:33.770026 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770008    2584 generic.go:356] "Generic (PLEG): container finished" podID="1b270190-1d31-44c3-9189-ee59c30ad802" containerID="e91d2506cd229043f5360fcf2decccab6bc9366c1ca5d06fcf207fcc6eefdd0d" exitCode=1
Jul 19 16:38:33.770090 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770069    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"f32a223599b73ddcfe1492f8bcace36c01ac55d3723448d6b2df14a1bed74609"}
Jul 19 16:38:33.770166 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770151    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"494294b565d722991eee71f5f14cd4115bb90fd3c53d8f657b9c1e9f830f561c"}
Jul 19 16:38:33.770219 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770205    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"620194e6c313830f5c86e98c45c23873114b84f65c013e02d865023af970f993"}
Jul 19 16:38:33.770271 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770259    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"0863251266d0bd8c236d46236c63997f4d5e303bd8914979bc2be68970debe04"}
Jul 19 16:38:33.770335 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770318    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerDied","Data":"e91d2506cd229043f5360fcf2decccab6bc9366c1ca5d06fcf207fcc6eefdd0d"}
Jul 19 16:38:33.770385 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.770349    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"7d72f09e8bd44235feef7cd57cc1ec5dd30810b2f1fdf2f34519d956bf1ee9c4"}
Jul 19 16:38:33.771492 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.771463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qbv9q" event={"ID":"900bb980-a550-45b4-a0ee-06d3e4c3f2d4","Type":"ContainerStarted","Data":"5d3d1fb9fa3dc5e6a7db5706d6180c0384b309da7c0348a44e4ffa2061e4012d"}
Jul 19 16:38:33.772794 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.772775    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9gfs" event={"ID":"2a4ba7cb-3517-4dd2-9442-8b050f343616","Type":"ContainerStarted","Data":"bc4d98b66a6132e6dd04d67fe452c84b8ca33f6b88ae147bc114ac9c355c1ecb"}
Jul 19 16:38:33.774098 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.774077    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2g5qt" event={"ID":"df49f583-c7e0-4367-af64-995e7bccf5ec","Type":"ContainerStarted","Data":"15ed3daa0a4ec398a44d9afa0a738c92ec21d6d581063416b24d1378cb542028"}
Jul 19 16:38:33.775442 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.775420    2584 generic.go:356] "Generic (PLEG): container finished" podID="79070897-558c-4406-a143-39cf00b7a155" containerID="eab8b6b788f33b6c8f593b40048cc3d1395c06dec43905d9585388bd532dd7d3" exitCode=0
Jul 19 16:38:33.775524 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.775468    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerDied","Data":"eab8b6b788f33b6c8f593b40048cc3d1395c06dec43905d9585388bd532dd7d3"}
Jul 19 16:38:33.784886 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.784852    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mbbtj" podStartSLOduration=4.809462601 podStartE2EDuration="23.784838844s" totalImagesPullingTime="18.975376243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:33.784758699 +0000 UTC m=+23.690010124" watchObservedRunningTime="2026-07-19 16:38:33.784838844 +0000 UTC m=+23.690090321"
Jul 19 16:38:33.797346 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.797305    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qbv9q" podStartSLOduration=4.84622752 podStartE2EDuration="23.797291191s" totalImagesPullingTime="18.951063671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:33.796880148 +0000 UTC m=+23.702131572" watchObservedRunningTime="2026-07-19 16:38:33.797291191 +0000 UTC m=+23.702542605"
Jul 19 16:38:33.811288 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.811243    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n9gfs" podStartSLOduration=4.854627425 podStartE2EDuration="23.811229886s" totalImagesPullingTime="18.956602461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:33.810861126 +0000 UTC m=+23.716112772" watchObservedRunningTime="2026-07-19 16:38:33.811229886 +0000 UTC m=+23.716481309"
Jul 19 16:38:33.824259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:33.824219    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2g5qt" podStartSLOduration=4.87749749 podStartE2EDuration="23.824205503s" totalImagesPullingTime="18.946708013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:33.823801659 +0000 UTC m=+23.729053082" watchObservedRunningTime="2026-07-19 16:38:33.824205503 +0000 UTC m=+23.729456926"
Jul 19 16:38:34.054429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.054403    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 19 16:38:34.596426 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.596313    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T16:38:34.054422959Z","UUID":"24f65246-0ef0-417d-a7ed-1618496412fe","Handler":null,"Name":"","Endpoint":""}
Jul 19 16:38:34.600212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.600181    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 19 16:38:34.600393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.600226    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 19 16:38:34.626661 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.626530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:34.626815 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:34.626656    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:34.626815 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.626698    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:34.626815 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:34.626786    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:34.627211 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.627051    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:34.627211 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:34.627162    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:34.779991 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.779955    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j9f75" event={"ID":"1072ee5d-843c-4c96-967f-240ac7e610ef","Type":"ContainerStarted","Data":"4e0cee281f65d86d15a340c36a8f6359b263fac75ded5f189fcb6137e3913c7e"}
Jul 19 16:38:34.782010 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.781984    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr" event={"ID":"1d61ddf7-7492-42b9-9c8b-4f69c4838231","Type":"ContainerStarted","Data":"ba2e5eb8f6a1903b6fe12f7f615042f3940f274cda7fad418a0b60929c3ccd2f"}
Jul 19 16:38:34.793956 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:34.793917    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j9f75" podStartSLOduration=5.840685961 podStartE2EDuration="24.7939044s" totalImagesPullingTime="18.953218439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:34.793884997 +0000 UTC m=+24.699136417" watchObservedRunningTime="2026-07-19 16:38:34.7939044 +0000 UTC m=+24.699155823"
Jul 19 16:38:35.788039 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:35.787463    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr" event={"ID":"1d61ddf7-7492-42b9-9c8b-4f69c4838231","Type":"ContainerStarted","Data":"c4a445fdb7d308987f9954a44f5b523cf52aa49cf772495e54a35d2e086a3d4d"}
Jul 19 16:38:35.790870 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:35.790849    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:38:35.791437 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:35.791414    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"9c8e386ff7d594bd8e002b7321b82d50f862e77e2b674e00fdfa8f9bd5f49cf4"}
Jul 19 16:38:35.807733 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:35.805962    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfcnr" podStartSLOduration=4.977989993 podStartE2EDuration="25.805939722s" totalImagesPullingTime="20.827949729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:35.804840416 +0000 UTC m=+25.710091853" watchObservedRunningTime="2026-07-19 16:38:35.805939722 +0000 UTC m=+25.711191147"
Jul 19 16:38:36.505205 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.505169    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:36.505820 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.505798    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:36.626738 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.626703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:36.626919 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.626703    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:36.626919 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:36.626828    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:36.626919 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.626862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:36.627085 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:36.626946    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:36.627085 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:36.627022    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:36.794349 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.794006    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:36.794780 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:36.794664    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-24cj4"
Jul 19 16:38:38.626733 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.626698    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:38.627533 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.626709    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:38.627533 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:38.626805    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:38.627533 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.626714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:38.627533 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:38.626890    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:38.627533 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:38.626994    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:38.801260 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.801227    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:38:38.801649 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.801625    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"1518499f16be589f95c26f298d00abe9b0cd13a562e0d4663050c3a9c603a012"}
Jul 19 16:38:38.801881 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.801858    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:38.802073 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.802054    2584 scope.go:117] "RemoveContainer" containerID="e91d2506cd229043f5360fcf2decccab6bc9366c1ca5d06fcf207fcc6eefdd0d"
Jul 19 16:38:38.803821 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.803592    2584 generic.go:356] "Generic (PLEG): container finished" podID="79070897-558c-4406-a143-39cf00b7a155" containerID="8fb95676ff4d61458003450f6797d8433ce20f62f663143aeed2194d4bd0de19" exitCode=0
Jul 19 16:38:38.803920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.803670    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerDied","Data":"8fb95676ff4d61458003450f6797d8433ce20f62f663143aeed2194d4bd0de19"}
Jul 19 16:38:38.820149 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.820124    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:38.832251 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.832226    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:38.914018 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:38.913920    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:39.810685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.810604    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:38:39.811122 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.811030    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" event={"ID":"1b270190-1d31-44c3-9189-ee59c30ad802","Type":"ContainerStarted","Data":"f087e274a9e5fb68239ec31392f5ed2c0d64b606c35d64430bbdfd25b7e0d941"}
Jul 19 16:38:39.811344 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.811327    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:39.811431 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.811398    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:39.813537 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.813509    2584 generic.go:356] "Generic (PLEG): container finished" podID="79070897-558c-4406-a143-39cf00b7a155" containerID="3ccd180800557090d2b3ef7c4619187293370eb4f7a753fc3e3ac355fc9f2c35" exitCode=0
Jul 19 16:38:39.813650 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.813553    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerDied","Data":"3ccd180800557090d2b3ef7c4619187293370eb4f7a753fc3e3ac355fc9f2c35"}
Jul 19 16:38:39.827995 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.827972    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:38:39.839654 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:39.839614    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2" podStartSLOduration=10.829380977 podStartE2EDuration="29.839598486s" totalImagesPullingTime="19.010217509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:39.837925054 +0000 UTC m=+29.743176479" watchObservedRunningTime="2026-07-19 16:38:39.839598486 +0000 UTC m=+29.744849911"
Jul 19 16:38:40.303985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.303955    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-46r78"]
Jul 19 16:38:40.304159 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.304092    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:40.304218 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:40.304177    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:40.307235 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.306778    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xqq8r"]
Jul 19 16:38:40.307235 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.306904    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:40.307235 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:40.307022    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:40.307833 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.307623    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pmwch"]
Jul 19 16:38:40.307833 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.307729    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:40.307833 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:40.307805    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:40.817535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.817453    2584 generic.go:356] "Generic (PLEG): container finished" podID="79070897-558c-4406-a143-39cf00b7a155" containerID="7c76d1c298af25836167e9caf5435a0d5cc0f96c89b96a2d8ae188994b97ca74" exitCode=0
Jul 19 16:38:40.817973 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:40.817548    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerDied","Data":"7c76d1c298af25836167e9caf5435a0d5cc0f96c89b96a2d8ae188994b97ca74"}
Jul 19 16:38:41.626797 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:41.626771    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:41.626930 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:41.626904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:41.627320 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:41.627301    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:41.627421 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:41.627400    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:42.626919 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:42.626687    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:42.627396 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:42.626966    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:43.200947 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:43.200906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:43.301230 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:43.301185    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:43.627069 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:43.626997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:43.627524 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:43.626997    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:43.627524 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:43.627140    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:43.627524 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:43.627219    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:44.275268 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:44.275231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:44.275448 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.275401    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:44.275503 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.275466    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:39:16.27544992 +0000 UTC m=+66.180701322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 16:38:44.375757 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:44.375717    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:44.375943 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:44.375818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:44.375943 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.375869    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:44.375943 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.375929    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret podName:c988c3e2-f0e0-4c7e-a826-5ff128caa1ab nodeName:}" failed. No retries permitted until 2026-07-19 16:39:16.375914693 +0000 UTC m=+66.281166098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret") pod "global-pull-secret-syncer-pmwch" (UID: "c988c3e2-f0e0-4c7e-a826-5ff128caa1ab") : object "kube-system"/"original-pull-secret" not registered
Jul 19 16:38:44.375943 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.375941    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 16:38:44.376158 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.375957    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 16:38:44.376158 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.375970    2584 projected.go:196] Error preparing data for projected volume kube-api-access-54wnb for pod openshift-network-diagnostics/network-check-target-46r78: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:44.376158 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.376023    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb podName:553bddfb-9715-4b2f-80e4-385e2a52ee89 nodeName:}" failed. No retries permitted until 2026-07-19 16:39:16.376007671 +0000 UTC m=+66.281259075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-54wnb" (UniqueName: "kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb") pod "network-check-target-46r78" (UID: "553bddfb-9715-4b2f-80e4-385e2a52ee89") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 16:38:44.626473 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:44.626390    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:44.626645 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:44.626520    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:38:45.626434 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:45.626343    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:45.627029 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:45.626348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:45.627029 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:45.626471    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pmwch" podUID="c988c3e2-f0e0-4c7e-a826-5ff128caa1ab"
Jul 19 16:38:45.627029 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:45.626519    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-46r78" podUID="553bddfb-9715-4b2f-80e4-385e2a52ee89"
Jul 19 16:38:45.834900 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:45.834862    2584 generic.go:356] "Generic (PLEG): container finished" podID="79070897-558c-4406-a143-39cf00b7a155" containerID="f811cd5d80b01b9b37afeda4ed742b1040d1f1ff3b029139fcb4c160f982b2fe" exitCode=0
Jul 19 16:38:45.835101 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:45.834920    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerDied","Data":"f811cd5d80b01b9b37afeda4ed742b1040d1f1ff3b029139fcb4c160f982b2fe"}
Jul 19 16:38:46.258027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.257999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:46.356094 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.356052    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:46.396381 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.396358    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-78.ec2.internal" event="NodeReady"
Jul 19 16:38:46.396525 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.396500    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 19 16:38:46.442623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.442588    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rcgfq"]
Jul 19 16:38:46.463340 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.463311    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cg26t"]
Jul 19 16:38:46.463501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.463438    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.466018 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.465989    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 19 16:38:46.466150 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.466030    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pk4kh\""
Jul 19 16:38:46.466150 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.466050    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 19 16:38:46.489384 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.489311    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cg26t"]
Jul 19 16:38:46.489384 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.489354    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcgfq"]
Jul 19 16:38:46.489573 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.489417    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.492009 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.491960    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 19 16:38:46.492133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.492028    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wcrqr\""
Jul 19 16:38:46.492133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.492034    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 19 16:38:46.492133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.492088    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 19 16:38:46.552779 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.552744    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:46.592915 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.592879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a8fa31d-fd58-4871-8a13-8fcfb27adcbb-cert\") pod \"ingress-canary-cg26t\" (UID: \"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb\") " pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.593064 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.592942    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjgt\" (UniqueName: \"kubernetes.io/projected/89cdf330-d376-49a0-9009-1de1be38456a-kube-api-access-7qjgt\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.593064 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.592971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.593064 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.593050    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89cdf330-d376-49a0-9009-1de1be38456a-tmp-dir\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.593189 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.593097    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98d8p\" (UniqueName: \"kubernetes.io/projected/0a8fa31d-fd58-4871-8a13-8fcfb27adcbb-kube-api-access-98d8p\") pod \"ingress-canary-cg26t\" (UID: \"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb\") " pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.593189 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.593116    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89cdf330-d376-49a0-9009-1de1be38456a-config-volume\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.626903 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.626868    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:38:46.629327 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.629306    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 19 16:38:46.629461 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.629356    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkhnw\""
Jul 19 16:38:46.694125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.694318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694146    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89cdf330-d376-49a0-9009-1de1be38456a-tmp-dir\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.694318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694172    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98d8p\" (UniqueName: \"kubernetes.io/projected/0a8fa31d-fd58-4871-8a13-8fcfb27adcbb-kube-api-access-98d8p\") pod \"ingress-canary-cg26t\" (UID: \"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb\") " pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.694318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89cdf330-d376-49a0-9009-1de1be38456a-config-volume\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.694318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694217    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a8fa31d-fd58-4871-8a13-8fcfb27adcbb-cert\") pod \"ingress-canary-cg26t\" (UID: \"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb\") " pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.694318 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:46.694243    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:38:46.694318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjgt\" (UniqueName: \"kubernetes.io/projected/89cdf330-d376-49a0-9009-1de1be38456a-kube-api-access-7qjgt\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.694627 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:46.694344    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:38:47.194319936 +0000 UTC m=+37.099571341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:38:46.694688 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694644    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89cdf330-d376-49a0-9009-1de1be38456a-tmp-dir\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.695002 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.694984    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/89cdf330-d376-49a0-9009-1de1be38456a-config-volume\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.698720 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.698698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a8fa31d-fd58-4871-8a13-8fcfb27adcbb-cert\") pod \"ingress-canary-cg26t\" (UID: \"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb\") " pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.704808 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.704782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjgt\" (UniqueName: \"kubernetes.io/projected/89cdf330-d376-49a0-9009-1de1be38456a-kube-api-access-7qjgt\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:46.705438 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.705418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98d8p\" (UniqueName: \"kubernetes.io/projected/0a8fa31d-fd58-4871-8a13-8fcfb27adcbb-kube-api-access-98d8p\") pod \"ingress-canary-cg26t\" (UID: \"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb\") " pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.797980 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.797900    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cg26t"
Jul 19 16:38:46.840831 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.840792    2584 generic.go:356] "Generic (PLEG): container finished" podID="79070897-558c-4406-a143-39cf00b7a155" containerID="fdbc4ddd3bae5584cc670d06dae21c5a1e351a398637b6dda5fb7bf2abd6b59d" exitCode=0
Jul 19 16:38:46.840964 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:46.840854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerDied","Data":"fdbc4ddd3bae5584cc670d06dae21c5a1e351a398637b6dda5fb7bf2abd6b59d"}
Jul 19 16:38:47.021389 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.021181    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cg26t"]
Jul 19 16:38:47.024816 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:38:47.024793    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8fa31d_fd58_4871_8a13_8fcfb27adcbb.slice/crio-6fb9f3ee1d528403c74f8327e0f4ce9da37c8e11ee81754b0675f728ff3c8f82 WatchSource:0}: Error finding container 6fb9f3ee1d528403c74f8327e0f4ce9da37c8e11ee81754b0675f728ff3c8f82: Status 404 returned error can't find the container with id 6fb9f3ee1d528403c74f8327e0f4ce9da37c8e11ee81754b0675f728ff3c8f82
Jul 19 16:38:47.198142 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.198111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:47.198351 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:47.198225    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:38:47.198351 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:47.198297    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:38:48.198261935 +0000 UTC m=+38.103513337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:38:47.627247 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.627163    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:38:47.627891 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.627167    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:38:47.629726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.629697    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 19 16:38:47.629726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.629723    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 19 16:38:47.630724 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.630703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sh2cq\""
Jul 19 16:38:47.630844 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.630721    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 19 16:38:47.847961 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.847926    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" event={"ID":"79070897-558c-4406-a143-39cf00b7a155","Type":"ContainerStarted","Data":"b98480404f845ca4192ff40807994542db101d336293d098ff9ed05352d51cdf"}
Jul 19 16:38:47.849065 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.849039    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cg26t" event={"ID":"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb","Type":"ContainerStarted","Data":"6fb9f3ee1d528403c74f8327e0f4ce9da37c8e11ee81754b0675f728ff3c8f82"}
Jul 19 16:38:47.870882 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.870832    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xrzwf" podStartSLOduration=9.060812835 podStartE2EDuration="37.870815838s" totalImagesPullingTime="28.810003003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:47.869914951 +0000 UTC m=+37.775166375" watchObservedRunningTime="2026-07-19 16:38:47.870815838 +0000 UTC m=+37.776067261"
Jul 19 16:38:47.992078 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:47.992042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:48.094122 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:48.094086    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:48.192854 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:48.192818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:48.207872 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:48.207838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:48.208052 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:48.207991    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:38:48.208106 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:48.208056    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:38:50.208038849 +0000 UTC m=+40.113290258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:38:48.293621 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:48.293527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:48.936805 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:48.936768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:49.332743 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:49.332663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:49.386817 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:49.386787    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:49.855093 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:49.854916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cg26t" event={"ID":"0a8fa31d-fd58-4871-8a13-8fcfb27adcbb","Type":"ContainerStarted","Data":"68196d600e6fd279542c529757b6d8b59daf1ff14996fd12ee18da49eaf2e14a"}
Jul 19 16:38:49.869210 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:49.869166    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cg26t" podStartSLOduration=1.468057624 podStartE2EDuration="3.869150657s" totalImagesPullingTime="2.401093033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:38:49.868424197 +0000 UTC m=+39.773675620" watchObservedRunningTime="2026-07-19 16:38:49.869150657 +0000 UTC m=+39.774402112"
Jul 19 16:38:49.873137 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:49.873111    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:50.223814 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:50.223781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:50.224188 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:50.223904    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:38:50.224188 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:50.223956    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:38:54.223942267 +0000 UTC m=+44.129193668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:38:51.099040 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:51.098971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:51.152657 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:51.152620    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:53.055650 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:53.055616    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:53.108137 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:53.108105    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:38:54.257405 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:38:54.257365    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:38:54.257801 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:54.257486    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:38:54.257801 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:38:54.257541    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:39:02.257525938 +0000 UTC m=+52.162777340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:39:00.539830 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:00.539786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:00.640151 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:00.640117    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:02.313937 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:02.313902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:39:02.314352 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:02.314068    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:39:02.314352 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:02.314136    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:39:18.314118872 +0000 UTC m=+68.219370273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:39:04.030065 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:04.030029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:04.131026 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:04.130986    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:04.746957 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:04.746923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:05.244761 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:05.244733    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:05.301171 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:05.301137    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:10.531594 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:10.531553    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:11.831374 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:11.831349    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b4lp2"
Jul 19 16:39:11.979786 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:11.979743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:12.079947 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:12.079912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:16.316055 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.316011    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:39:16.318918 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.318899    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 19 16:39:16.326848 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:16.326830    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 19 16:39:16.326913 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:16.326906    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:40:20.326885235 +0000 UTC m=+130.232136636 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : secret "metrics-daemon-secret" not found
Jul 19 16:39:16.416827 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.416793    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:39:16.416827 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.416831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:39:16.419408 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.419385    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 19 16:39:16.419473 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.419416    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 19 16:39:16.429340 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.429321    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 19 16:39:16.431182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.431164    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c988c3e2-f0e0-4c7e-a826-5ff128caa1ab-original-pull-secret\") pod \"global-pull-secret-syncer-pmwch\" (UID: \"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab\") " pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:39:16.440571 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.440550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54wnb\" (UniqueName: \"kubernetes.io/projected/553bddfb-9715-4b2f-80e4-385e2a52ee89-kube-api-access-54wnb\") pod \"network-check-target-46r78\" (UID: \"553bddfb-9715-4b2f-80e4-385e2a52ee89\") " pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:39:16.445353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.445333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pmwch"
Jul 19 16:39:16.599368 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.599271    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pmwch"]
Jul 19 16:39:16.602403 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:39:16.602377    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc988c3e2_f0e0_4c7e_a826_5ff128caa1ab.slice/crio-1639847621b410cb1eee173950b1f5bcf08ee97aa763eaf53a55bd64360614ff WatchSource:0}: Error finding container 1639847621b410cb1eee173950b1f5bcf08ee97aa763eaf53a55bd64360614ff: Status 404 returned error can't find the container with id 1639847621b410cb1eee173950b1f5bcf08ee97aa763eaf53a55bd64360614ff
Jul 19 16:39:16.742416 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.742386    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sh2cq\""
Jul 19 16:39:16.750742 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.750720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:39:16.906356 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.906315    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-46r78"]
Jul 19 16:39:16.909635 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:39:16.909609    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553bddfb_9715_4b2f_80e4_385e2a52ee89.slice/crio-60fcacd2988ccd8c25a8a2a7a6c1ec851c0712a1a750c7f1a47e2469b312a67e WatchSource:0}: Error finding container 60fcacd2988ccd8c25a8a2a7a6c1ec851c0712a1a750c7f1a47e2469b312a67e: Status 404 returned error can't find the container with id 60fcacd2988ccd8c25a8a2a7a6c1ec851c0712a1a750c7f1a47e2469b312a67e
Jul 19 16:39:16.918857 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.918834    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-46r78" event={"ID":"553bddfb-9715-4b2f-80e4-385e2a52ee89","Type":"ContainerStarted","Data":"60fcacd2988ccd8c25a8a2a7a6c1ec851c0712a1a750c7f1a47e2469b312a67e"}
Jul 19 16:39:16.919961 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:16.919939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pmwch" event={"ID":"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab","Type":"ContainerStarted","Data":"1639847621b410cb1eee173950b1f5bcf08ee97aa763eaf53a55bd64360614ff"}
Jul 19 16:39:18.329960 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:18.329919    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:39:18.330466 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:18.330071    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:39:18.330466 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:18.330133    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:39:50.330115848 +0000 UTC m=+100.235367249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:39:21.934772 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:21.934735    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pmwch" event={"ID":"c988c3e2-f0e0-4c7e-a826-5ff128caa1ab","Type":"ContainerStarted","Data":"322eb3fed015e4e14b4f130f11b12503cd688be7f51ce43a05023e73e66a9c3b"}
Jul 19 16:39:21.936222 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:21.936194    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-46r78" event={"ID":"553bddfb-9715-4b2f-80e4-385e2a52ee89","Type":"ContainerStarted","Data":"f993f6fc3527d01267545e81fd9986f10dc7c08f1e0ffd1085cc7f385781fe51"}
Jul 19 16:39:21.936367 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:21.936327    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:39:21.949460 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:21.949413    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pmwch" podStartSLOduration=67.165370173 podStartE2EDuration="1m11.949400432s" totalImagesPullingTime="4.784030259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:39:21.949059439 +0000 UTC m=+71.854310864" watchObservedRunningTime="2026-07-19 16:39:21.949400432 +0000 UTC m=+71.854651855"
Jul 19 16:39:21.963643 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:21.963598    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-46r78" podStartSLOduration=67.490762128 podStartE2EDuration="1m11.963584439s" totalImagesPullingTime="4.472822311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:39:21.96295646 +0000 UTC m=+71.868207886" watchObservedRunningTime="2026-07-19 16:39:21.963584439 +0000 UTC m=+71.868835863"
Jul 19 16:39:25.991778 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:25.991743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:26.091815 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:26.091780    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:39:50.341474 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:50.341417    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:39:50.341902 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:50.341567    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:39:50.341902 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:39:50.341634    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:40:54.341619211 +0000 UTC m=+164.246870613 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:39:52.940880 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:39:52.940848    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-46r78"
Jul 19 16:40:05.685670 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.685635    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"]
Jul 19 16:40:05.687472 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.687454    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.689979 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.689963    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 19 16:40:05.691042 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.691029    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 19 16:40:05.691188 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.691177    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 19 16:40:05.691708 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.691695    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 19 16:40:05.701154 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.701127    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"]
Jul 19 16:40:05.714243 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.714216    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"]
Jul 19 16:40:05.716120 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.716105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.719017 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.718985    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 19 16:40:05.720329 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.719329    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 19 16:40:05.720329 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.719358    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 19 16:40:05.720329 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.719677    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 19 16:40:05.726312 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.726268    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"]
Jul 19 16:40:05.746123 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.746092    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a945bac3-3677-480f-ba36-8fe1d8e4012d-tmp\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.746123 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.746124    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a945bac3-3677-480f-ba36-8fe1d8e4012d-klusterlet-config\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.746343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.746149    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858xv\" (UniqueName: \"kubernetes.io/projected/a945bac3-3677-480f-ba36-8fe1d8e4012d-kube-api-access-858xv\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.846464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-858xv\" (UniqueName: \"kubernetes.io/projected/a945bac3-3677-480f-ba36-8fe1d8e4012d-kube-api-access-858xv\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.846592 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.846592 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846495    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.846592 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-hub\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.846737 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846602    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tq9\" (UniqueName: \"kubernetes.io/projected/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-kube-api-access-89tq9\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.846737 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846630    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.846737 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846655    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-ca\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.846737 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846686    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a945bac3-3677-480f-ba36-8fe1d8e4012d-tmp\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.846737 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.846705    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a945bac3-3677-480f-ba36-8fe1d8e4012d-klusterlet-config\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.847053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.847031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a945bac3-3677-480f-ba36-8fe1d8e4012d-tmp\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.849221 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.849202    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a945bac3-3677-480f-ba36-8fe1d8e4012d-klusterlet-config\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.854686 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.854662    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-858xv\" (UniqueName: \"kubernetes.io/projected/a945bac3-3677-480f-ba36-8fe1d8e4012d-kube-api-access-858xv\") pod \"klusterlet-addon-workmgr-855bcdd59f-pg8rx\" (UID: \"a945bac3-3677-480f-ba36-8fe1d8e4012d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:05.947392 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.947303    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-ca\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.947392 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.947342    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.947392 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.947364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.947598 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.947531    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-hub\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.947598 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.947565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tq9\" (UniqueName: \"kubernetes.io/projected/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-kube-api-access-89tq9\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.947690 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.947608    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.948142 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.948048    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.949957 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.949929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-hub\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.950078 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.949985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.950439 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.950419    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-ca\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.950568 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.950553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.956394 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.956375    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tq9\" (UniqueName: \"kubernetes.io/projected/51b12b1b-fec6-4422-a0a5-1a4c1bea993d-kube-api-access-89tq9\") pod \"cluster-proxy-proxy-agent-76cbc9bb8b-dksb7\" (UID: \"51b12b1b-fec6-4422-a0a5-1a4c1bea993d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:05.996556 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:05.996517    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:06.039384 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:06.039349    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:40:06.127464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:06.127433    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"]
Jul 19 16:40:06.131541 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:40:06.131502    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda945bac3_3677_480f_ba36_8fe1d8e4012d.slice/crio-88e7561b4a839b3c5893d9a8b327fa3f2d288903a0d0819e41b2399a5daa1e98 WatchSource:0}: Error finding container 88e7561b4a839b3c5893d9a8b327fa3f2d288903a0d0819e41b2399a5daa1e98: Status 404 returned error can't find the container with id 88e7561b4a839b3c5893d9a8b327fa3f2d288903a0d0819e41b2399a5daa1e98
Jul 19 16:40:06.176017 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:06.175987    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"]
Jul 19 16:40:06.179426 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:40:06.179402    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51b12b1b_fec6_4422_a0a5_1a4c1bea993d.slice/crio-ccd049c2a94d79f42fcbfea954045e5347c219e6b35bf9912fe0b749b79e10f1 WatchSource:0}: Error finding container ccd049c2a94d79f42fcbfea954045e5347c219e6b35bf9912fe0b749b79e10f1: Status 404 returned error can't find the container with id ccd049c2a94d79f42fcbfea954045e5347c219e6b35bf9912fe0b749b79e10f1
Jul 19 16:40:07.047734 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:07.047647    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" event={"ID":"51b12b1b-fec6-4422-a0a5-1a4c1bea993d","Type":"ContainerStarted","Data":"ccd049c2a94d79f42fcbfea954045e5347c219e6b35bf9912fe0b749b79e10f1"}
Jul 19 16:40:07.052698 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:07.052537    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx" event={"ID":"a945bac3-3677-480f-ba36-8fe1d8e4012d","Type":"ContainerStarted","Data":"88e7561b4a839b3c5893d9a8b327fa3f2d288903a0d0819e41b2399a5daa1e98"}
Jul 19 16:40:12.069985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.069943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" event={"ID":"51b12b1b-fec6-4422-a0a5-1a4c1bea993d","Type":"ContainerStarted","Data":"e252810d2f4cb02c537e14360ef53697070d97ab0f9debcb2649c1f1282e1eff"}
Jul 19 16:40:12.069985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.069996    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" event={"ID":"51b12b1b-fec6-4422-a0a5-1a4c1bea993d","Type":"ContainerStarted","Data":"76c4dbfdfe39b85b3e97d713f80b02ec9f9f65827014eb0b9f416cfee561b1e2"}
Jul 19 16:40:12.070540 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.070008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" event={"ID":"51b12b1b-fec6-4422-a0a5-1a4c1bea993d","Type":"ContainerStarted","Data":"c07519ebf76b52141e400381491a393d7edd7ebbac46a2c0a6b77fc13a97c04d"}
Jul 19 16:40:12.071318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.071294    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx" event={"ID":"a945bac3-3677-480f-ba36-8fe1d8e4012d","Type":"ContainerStarted","Data":"57ad404e012b9507feab4093773e4966eb3424c8988300c5698bdfd511880711"}
Jul 19 16:40:12.071518 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.071501    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:12.073041 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.073023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:40:12.089705 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.089655    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" podStartSLOduration=1.9494703580000001 podStartE2EDuration="7.089640883s" totalImagesPullingTime="5.140170525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:40:12.088441454 +0000 UTC m=+121.993692878" watchObservedRunningTime="2026-07-19 16:40:12.089640883 +0000 UTC m=+121.994892324"
Jul 19 16:40:12.106040 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:12.105995    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx" podStartSLOduration=1.918373669 podStartE2EDuration="7.105980447s" totalImagesPullingTime="5.187606778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:40:12.105784726 +0000 UTC m=+122.011036150" watchObservedRunningTime="2026-07-19 16:40:12.105980447 +0000 UTC m=+122.011231871"
Jul 19 16:40:20.354575 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:20.354532    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:40:20.354971 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:40:20.354695    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 19 16:40:20.354971 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:40:20.354771    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:42:22.354755451 +0000 UTC m=+252.260006855 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : secret "metrics-daemon-secret" not found
Jul 19 16:40:49.472252 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:40:49.472208    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rcgfq" podUID="89cdf330-d376-49a0-9009-1de1be38456a"
Jul 19 16:40:49.635783 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:40:49.635728    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:40:50.176740 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:50.176711    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:40:54.397158 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:40:54.397111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:40:54.397599 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:40:54.397264    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 19 16:40:54.397599 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:40:54.397335    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls podName:89cdf330-d376-49a0-9009-1de1be38456a nodeName:}" failed. No retries permitted until 2026-07-19 16:42:56.397317663 +0000 UTC m=+286.302569064 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls") pod "dns-default-rcgfq" (UID: "89cdf330-d376-49a0-9009-1de1be38456a") : secret "dns-default-metrics-tls" not found
Jul 19 16:41:03.627078 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:41:03.626978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:42:01.997489 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:01.997457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:12.072548 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:12.072502    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx" podUID="a945bac3-3677-480f-ba36-8fe1d8e4012d" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jul 19 16:42:12.402033 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:12.402005    2584 generic.go:356] "Generic (PLEG): container finished" podID="a945bac3-3677-480f-ba36-8fe1d8e4012d" containerID="57ad404e012b9507feab4093773e4966eb3424c8988300c5698bdfd511880711" exitCode=1
Jul 19 16:42:12.402191 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:12.402048    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx" event={"ID":"a945bac3-3677-480f-ba36-8fe1d8e4012d","Type":"ContainerDied","Data":"57ad404e012b9507feab4093773e4966eb3424c8988300c5698bdfd511880711"}
Jul 19 16:42:12.402409 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:12.402391    2584 scope.go:117] "RemoveContainer" containerID="57ad404e012b9507feab4093773e4966eb3424c8988300c5698bdfd511880711"
Jul 19 16:42:13.406809 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:13.406771    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx" event={"ID":"a945bac3-3677-480f-ba36-8fe1d8e4012d","Type":"ContainerStarted","Data":"745293fcd3fe9ab47c669140bec21c141d62ace305027e345fbf20b487759ea4"}
Jul 19 16:42:13.407329 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:13.407044    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:42:13.407687 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:13.407671    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-855bcdd59f-pg8rx"
Jul 19 16:42:22.431169 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:22.431137    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:42:22.431600 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:22.431296    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 19 16:42:22.431600 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:22.431367    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs podName:3c09c151-0fa7-4b87-98b9-7b5cf9aedeff nodeName:}" failed. No retries permitted until 2026-07-19 16:44:24.431349634 +0000 UTC m=+374.336601036 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs") pod "network-metrics-daemon-xqq8r" (UID: "3c09c151-0fa7-4b87-98b9-7b5cf9aedeff") : secret "metrics-daemon-secret" not found
Jul 19 16:42:24.876654 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:24.876619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:24.977715 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:24.977672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:26.662230 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:26.662198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:26.715186 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:26.715155    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:28.637293 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:28.637250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.068174 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.068141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.173210 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.173182    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.249051 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.249023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.330516 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.330430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.389868 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.389839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.590114 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.590034    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pbbms"]
Jul 19 16:42:29.592293 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.592253    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.595177 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.595153    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 19 16:42:29.595322 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.595172    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 19 16:42:29.595322 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.595202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-z6jkb\""
Jul 19 16:42:29.596060 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.596046    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 19 16:42:29.596156 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.596071    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 19 16:42:29.606059 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.606039    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pbbms"]
Jul 19 16:42:29.638319 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.638293    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.685548 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.685501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.685548 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.685544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.685763 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.685595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-crio-socket\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.685763 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.685634    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-data-volume\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.685763 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.685668    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5ls\" (UniqueName: \"kubernetes.io/projected/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-api-access-wt5ls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.695979 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.695949    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:29.786194 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.786194 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786225    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-crio-socket\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-data-volume\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786297    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5ls\" (UniqueName: \"kubernetes.io/projected/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-api-access-wt5ls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:29.786307    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:29.786350    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786353    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-crio-socket\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:29.786397    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:30.28636636 +0000 UTC m=+260.191617762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:29.786464 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:29.786419    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:30.286409309 +0000 UTC m=+260.191660710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : configmap "kube-rbac-proxy" not found
Jul 19 16:42:29.786809 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.786608    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-data-volume\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:29.794654 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:29.794633    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5ls\" (UniqueName: \"kubernetes.io/projected/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-api-access-wt5ls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:30.290584 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:30.290549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:30.290584 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:30.290585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:30.290800 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:30.290658    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 19 16:42:30.290800 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:30.290697    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:30.290800 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:30.290717    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:31.290702122 +0000 UTC m=+261.195953524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : configmap "kube-rbac-proxy" not found
Jul 19 16:42:30.290800 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:30.290751    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:31.290737455 +0000 UTC m=+261.195988861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:30.970421 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:30.970388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:31.071176 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:31.071142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:31.298098 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:31.297985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:31.298098 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:31.298043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:31.298098 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:31.298107    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 19 16:42:31.298379 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:31.298155    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:31.298379 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:31.298159    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:33.298146094 +0000 UTC m=+263.203397495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : configmap "kube-rbac-proxy" not found
Jul 19 16:42:31.298379 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:31.298221    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:33.298206261 +0000 UTC m=+263.203457664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:31.867330 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:31.867301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:32.204334 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:32.204312    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2g5qt_df49f583-c7e0-4367-af64-995e7bccf5ec/dns-node-resolver/0.log"
Jul 19 16:42:33.310987 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:33.310953    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:33.310987 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:33.310991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:33.311497 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:33.311068    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 19 16:42:33.311497 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:33.311101    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:33.311497 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:33.311119    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:37.311105631 +0000 UTC m=+267.216357033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : configmap "kube-rbac-proxy" not found
Jul 19 16:42:33.311497 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:33.311179    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:37.311165356 +0000 UTC m=+267.216416759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:33.403879 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:33.403852    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qbv9q_900bb980-a550-45b4-a0ee-06d3e4c3f2d4/node-ca/0.log"
Jul 19 16:42:34.003751 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.003725    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cg26t_0a8fa31d-fd58-4871-8a13-8fcfb27adcbb/serve-healthcheck-canary/0.log"
Jul 19 16:42:34.137527 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.137491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:34.238428 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.238396    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:34.374899 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.374819    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:34.475914 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.475883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:34.576909 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.576876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:34.807641 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.807569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:34.964638 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:34.964595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:35.199087 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:35.199059    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:35.994164 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:35.994130    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:36.193491 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:36.193459    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:36.798517 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:36.798483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:37.201478 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:37.201447    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:37.343489 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:37.343449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:37.343489 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:37.343486    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:37.343691 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:37.343596    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:37.343691 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:37.343651    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:45.343636559 +0000 UTC m=+275.248887962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : secret "insights-runtime-extractor-tls" not found
Jul 19 16:42:37.343691 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:37.343599    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 19 16:42:37.343825 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:37.343718    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm podName:7dfa6b2b-b60a-4c4b-8585-083e98bf4204 nodeName:}" failed. No retries permitted until 2026-07-19 16:42:45.343707398 +0000 UTC m=+275.248958800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pbbms" (UID: "7dfa6b2b-b60a-4c4b-8585-083e98bf4204") : configmap "kube-rbac-proxy" not found
Jul 19 16:42:37.396349 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:37.396319    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:38.018403 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:38.018374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:38.118469 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:38.118435    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:38.369627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:38.369539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:38.597334 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:38.597302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:38.797652 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:38.797617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:39.199057 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:39.199024    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:39.395402 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:39.395367    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:39.880573 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:39.880539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:40.395928 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:40.395898    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:40.580922 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:40.580885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:40.799382 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:40.799348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:40.994851 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:40.994815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:41.402732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:41.402702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:41.999038 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:41.999000    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:42.339981 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:42.339892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:42.441091 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:42.441057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:42.598216 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:42.598127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:43.025519 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:43.025484    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:43.794419 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:43.794383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:43.995303 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:43.995250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:44.399748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:44.399714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:44.594917 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:44.594883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:44.791376 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:44.791267    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:45.409518 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:45.409484    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:45.409899 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:45.409526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:45.410017 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:45.409983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:45.411989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:45.411970    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7dfa6b2b-b60a-4c4b-8585-083e98bf4204-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pbbms\" (UID: \"7dfa6b2b-b60a-4c4b-8585-083e98bf4204\") " pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:45.501228 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:45.501194    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pbbms"
Jul 19 16:42:45.624957 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:45.624925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pbbms"]
Jul 19 16:42:45.627650 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:42:45.627626    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfa6b2b_b60a_4c4b_8585_083e98bf4204.slice/crio-e9ef66ba261905d00d37ed252f3d192dbb8808ce793bdc0a6367bf693ef21a92 WatchSource:0}: Error finding container e9ef66ba261905d00d37ed252f3d192dbb8808ce793bdc0a6367bf693ef21a92: Status 404 returned error can't find the container with id e9ef66ba261905d00d37ed252f3d192dbb8808ce793bdc0a6367bf693ef21a92
Jul 19 16:42:46.041094 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:46.041053    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" podUID="51b12b1b-fec6-4422-a0a5-1a4c1bea993d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 19 16:42:46.499082 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:46.498997    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pbbms" event={"ID":"7dfa6b2b-b60a-4c4b-8585-083e98bf4204","Type":"ContainerStarted","Data":"e9ef66ba261905d00d37ed252f3d192dbb8808ce793bdc0a6367bf693ef21a92"}
Jul 19 16:42:50.458797 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.458764    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67ffb6bd68-nwxns"]
Jul 19 16:42:50.462138 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.462114    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.464653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.464627    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 19 16:42:50.464965 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.464947    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 19 16:42:50.465093 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.465065    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qblxf\""
Jul 19 16:42:50.465176 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.465065    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 19 16:42:50.470601 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.470578    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 19 16:42:50.475577 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.475552    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67ffb6bd68-nwxns"]
Jul 19 16:42:50.522720 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.522682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:50.554264 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0703ff3-025a-49ee-91a4-2a8b64334a87-trusted-ca\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554264 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0703ff3-025a-49ee-91a4-2a8b64334a87-registry-certificates\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554368    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhv4\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-kube-api-access-wdhv4\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554431    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-registry-tls\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-bound-sa-token\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554485    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0703ff3-025a-49ee-91a4-2a8b64334a87-image-registry-private-configuration\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554639 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554508    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0703ff3-025a-49ee-91a4-2a8b64334a87-installation-pull-secrets\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.554639 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.554533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0703ff3-025a-49ee-91a4-2a8b64334a87-ca-trust-extracted\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.621833 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.621799    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:50.655838 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655812    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-registry-tls\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.655989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655853    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-bound-sa-token\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.655989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655877    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0703ff3-025a-49ee-91a4-2a8b64334a87-image-registry-private-configuration\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.655989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655901    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0703ff3-025a-49ee-91a4-2a8b64334a87-installation-pull-secrets\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.655989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0703ff3-025a-49ee-91a4-2a8b64334a87-ca-trust-extracted\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.655989 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0703ff3-025a-49ee-91a4-2a8b64334a87-trusted-ca\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.656225 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.655992    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0703ff3-025a-49ee-91a4-2a8b64334a87-registry-certificates\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.656225 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.656033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhv4\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-kube-api-access-wdhv4\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.656898 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.656869    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0703ff3-025a-49ee-91a4-2a8b64334a87-ca-trust-extracted\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.661535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.657307    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0703ff3-025a-49ee-91a4-2a8b64334a87-trusted-ca\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.661535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.657738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0703ff3-025a-49ee-91a4-2a8b64334a87-registry-certificates\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.661535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.659789    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0703ff3-025a-49ee-91a4-2a8b64334a87-installation-pull-secrets\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.663075 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.663022    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-registry-tls\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.664075 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.664054    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0703ff3-025a-49ee-91a4-2a8b64334a87-image-registry-private-configuration\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.668323 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.668300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhv4\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-kube-api-access-wdhv4\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.668601 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.668579    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0703ff3-025a-49ee-91a4-2a8b64334a87-bound-sa-token\") pod \"image-registry-67ffb6bd68-nwxns\" (UID: \"c0703ff3-025a-49ee-91a4-2a8b64334a87\") " pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.773326 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.773205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:50.902118 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:50.902083    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67ffb6bd68-nwxns"]
Jul 19 16:42:50.904795 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:42:50.904771    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0703ff3_025a_49ee_91a4_2a8b64334a87.slice/crio-e87dfd0211bf1e11fc4cf32b499bcd27f249c8be4fc9abe78055cd7297613baf WatchSource:0}: Error finding container e87dfd0211bf1e11fc4cf32b499bcd27f249c8be4fc9abe78055cd7297613baf: Status 404 returned error can't find the container with id e87dfd0211bf1e11fc4cf32b499bcd27f249c8be4fc9abe78055cd7297613baf
Jul 19 16:42:51.100753 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.100653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.199563 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.199505    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.299679 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.299624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.399987 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.399947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.519438 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.519353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pbbms" event={"ID":"7dfa6b2b-b60a-4c4b-8585-083e98bf4204","Type":"ContainerStarted","Data":"889a847dff74a8383d4e259b5613c5ea47dbf9f975c6eaa1466b3e28a02b5c2b"}
Jul 19 16:42:51.521389 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.521358    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns" event={"ID":"c0703ff3-025a-49ee-91a4-2a8b64334a87","Type":"ContainerStarted","Data":"4bcebf4a1ddf36088cf1490f4bc55e31c4bc8de3b92d337bbf59218e99d564db"}
Jul 19 16:42:51.521526 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.521411    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns" event={"ID":"c0703ff3-025a-49ee-91a4-2a8b64334a87","Type":"ContainerStarted","Data":"e87dfd0211bf1e11fc4cf32b499bcd27f249c8be4fc9abe78055cd7297613baf"}
Jul 19 16:42:51.521624 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.521606    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:42:51.545364 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.545315    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns" podStartSLOduration=1.5452981609999998 podStartE2EDuration="1.545298161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:42:51.544137081 +0000 UTC m=+281.449388504" watchObservedRunningTime="2026-07-19 16:42:51.545298161 +0000 UTC m=+281.450549583"
Jul 19 16:42:51.575991 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.575929    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.676245 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.676153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.776685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.776653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:51.877131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:51.877095    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.186490 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.186456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.286003 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.285966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.398393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.398358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.574230 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.574138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.774319 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.774268    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.874910 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.874787    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.945640 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.945584    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:52.999613 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:52.999579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:53.100465 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:53.100430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:53.177872 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:42:53.177829    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rcgfq" podUID="89cdf330-d376-49a0-9009-1de1be38456a"
Jul 19 16:42:53.200705 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:53.200675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:53.527833 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:53.527760    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:42:53.658645 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:53.658602    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:54.532419 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:54.532383    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pbbms" event={"ID":"7dfa6b2b-b60a-4c4b-8585-083e98bf4204","Type":"ContainerStarted","Data":"52fa389c35e6cf1bd3d6971452a9d8b9801cfcdf7f37de802c1c4e1178ffc851"}
Jul 19 16:42:54.596809 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:54.596768    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:54.652543 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:54.652506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:42:56.041374 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:56.041325    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" podUID="51b12b1b-fec6-4422-a0a5-1a4c1bea993d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 19 16:42:56.402057 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:56.402006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:42:56.404919 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:56.404892    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/89cdf330-d376-49a0-9009-1de1be38456a-metrics-tls\") pod \"dns-default-rcgfq\" (UID: \"89cdf330-d376-49a0-9009-1de1be38456a\") " pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:42:56.531221 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:56.531191    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pk4kh\""
Jul 19 16:42:56.539359 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:56.539321    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:42:57.681997 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:57.681925    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcgfq"]
Jul 19 16:42:57.684478 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:42:57.684452    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cdf330_d376_49a0_9009_1de1be38456a.slice/crio-d4bc59836c2e5c0ad3f1f6ad030ada292bdb648fc15e02fd2cfda6351670c99a WatchSource:0}: Error finding container d4bc59836c2e5c0ad3f1f6ad030ada292bdb648fc15e02fd2cfda6351670c99a: Status 404 returned error can't find the container with id d4bc59836c2e5c0ad3f1f6ad030ada292bdb648fc15e02fd2cfda6351670c99a
Jul 19 16:42:58.548627 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:58.548587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pbbms" event={"ID":"7dfa6b2b-b60a-4c4b-8585-083e98bf4204","Type":"ContainerStarted","Data":"323688e6eb210b14baf0d3b30f4191578a9f60528a5f6b2a174cb67d174a6a77"}
Jul 19 16:42:58.549820 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:58.549794    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcgfq" event={"ID":"89cdf330-d376-49a0-9009-1de1be38456a","Type":"ContainerStarted","Data":"d4bc59836c2e5c0ad3f1f6ad030ada292bdb648fc15e02fd2cfda6351670c99a"}
Jul 19 16:42:58.564754 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:58.564689    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pbbms" podStartSLOduration=17.705286524 podStartE2EDuration="29.564675132s" totalImagesPullingTime="11.859388608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 16:42:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:42:58.563916202 +0000 UTC m=+288.469167631" watchObservedRunningTime="2026-07-19 16:42:58.564675132 +0000 UTC m=+288.469926556"
Jul 19 16:42:59.554834 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:59.554736    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcgfq" event={"ID":"89cdf330-d376-49a0-9009-1de1be38456a","Type":"ContainerStarted","Data":"19e4074c760ab483439cada88132816beb22b81d6bbbd00e18267114f270ac0b"}
Jul 19 16:42:59.554834 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:59.554803    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcgfq" event={"ID":"89cdf330-d376-49a0-9009-1de1be38456a","Type":"ContainerStarted","Data":"f0484f179fd3f8f98bc8ad11da6c8b338ceb2c44f59610b3baa666ab842a23c7"}
Jul 19 16:42:59.555228 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:59.555050    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:42:59.571712 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:42:59.571662    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rcgfq" podStartSLOduration=251.953896255 podStartE2EDuration="4m13.571647305s" totalImagesPullingTime="1.61775105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:42:59.570160107 +0000 UTC m=+289.475411530" watchObservedRunningTime="2026-07-19 16:42:59.571647305 +0000 UTC m=+289.476898707"
Jul 19 16:43:00.675253 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:00.675213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:00.750372 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:00.750342    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:00.839474 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:00.839441    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:01.447211 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:01.447179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:02.415463 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:02.415433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:02.516440 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:02.516403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:06.041101 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.041059    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" podUID="51b12b1b-fec6-4422-a0a5-1a4c1bea993d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 19 16:43:06.041101 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.041135    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7"
Jul 19 16:43:06.041812 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.041754    2584 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"e252810d2f4cb02c537e14360ef53697070d97ab0f9debcb2649c1f1282e1eff"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 19 16:43:06.041958 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.041826    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" podUID="51b12b1b-fec6-4422-a0a5-1a4c1bea993d" containerName="service-proxy" containerID="cri-o://e252810d2f4cb02c537e14360ef53697070d97ab0f9debcb2649c1f1282e1eff" gracePeriod=30
Jul 19 16:43:06.579481 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.579448    2584 generic.go:356] "Generic (PLEG): container finished" podID="51b12b1b-fec6-4422-a0a5-1a4c1bea993d" containerID="e252810d2f4cb02c537e14360ef53697070d97ab0f9debcb2649c1f1282e1eff" exitCode=2
Jul 19 16:43:06.579659 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.579495    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" event={"ID":"51b12b1b-fec6-4422-a0a5-1a4c1bea993d","Type":"ContainerDied","Data":"e252810d2f4cb02c537e14360ef53697070d97ab0f9debcb2649c1f1282e1eff"}
Jul 19 16:43:06.579659 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.579529    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76cbc9bb8b-dksb7" event={"ID":"51b12b1b-fec6-4422-a0a5-1a4c1bea993d","Type":"ContainerStarted","Data":"440e7318ef6bc0ecdb246441d3196ecdf28073dc326e55c5411b30ed51620fb2"}
Jul 19 16:43:06.596598 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:06.596564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:06.627730 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:43:06.627689    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xqq8r" podUID="3c09c151-0fa7-4b87-98b9-7b5cf9aedeff"
Jul 19 16:43:09.560300 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:09.560255    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rcgfq"
Jul 19 16:43:10.202015 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:10.201978    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:10.524404 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:10.524333    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:43:10.529647 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:10.529627    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 19 16:43:10.530117 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:10.530098    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:43:10.777594 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:10.777515    2584 patch_prober.go:28] interesting pod/image-registry-67ffb6bd68-nwxns container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 19 16:43:10.777594 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:10.777567    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns" podUID="c0703ff3-025a-49ee-91a4-2a8b64334a87" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 19 16:43:12.118044 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:12.118005    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:12.202523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:12.202491    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:12.529131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:12.529105    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67ffb6bd68-nwxns"
Jul 19 16:43:12.692855 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:12.692825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:12.816861 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:12.816783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:12.980757 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:12.980726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:13.048079 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:13.048048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:13.148747 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:13.148672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:14.655686 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:14.655656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:14.708598 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:14.708567    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:15.109486 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:15.109399    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:15.409789 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:15.409761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:15.510784 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:15.510749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:15.567542 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:15.567506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.007952 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.007918    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.228780 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.228745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.281878 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.281799    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l9qfx"]
Jul 19 16:43:17.328096 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.328069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.395508 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.395478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.507141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.507113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.617917 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.617829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.664198 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.664164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.666798 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.666776    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 19 16:43:17.666798 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.666794    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 19 16:43:17.667001 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.666851    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 19 16:43:17.667001 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.666909    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 19 16:43:17.667001 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.666975    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5kc4x\""
Jul 19 16:43:17.667829 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.667815    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 19 16:43:17.667829 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.667822    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 19 16:43:17.718446 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.718411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.759064 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-wtmp\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-sys\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-root\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-textfile\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759413 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759251    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-tls\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759413 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759314    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d1736-1ef0-428e-866b-3f8f796c4471-metrics-client-ca\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759413 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759343    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759413 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759375    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.759413 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.759401    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26qj\" (UniqueName: \"kubernetes.io/projected/8a8d1736-1ef0-428e-866b-3f8f796c4471-kube-api-access-r26qj\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.819429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.819399    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.860096 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860069    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-wtmp\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-sys\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860135    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-root\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-textfile\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-tls\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-sys\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860232    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d1736-1ef0-428e-866b-3f8f796c4471-metrics-client-ca\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860507 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-wtmp\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860507 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860507 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860304    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860507 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860329    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26qj\" (UniqueName: \"kubernetes.io/projected/8a8d1736-1ef0-428e-866b-3f8f796c4471-kube-api-access-r26qj\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860507 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860258    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8a8d1736-1ef0-428e-866b-3f8f796c4471-root\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860745 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-textfile\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.860937 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860917    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d1736-1ef0-428e-866b-3f8f796c4471-metrics-client-ca\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.861007 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.860941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-accelerators-collector-config\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.862738 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.862707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.862859 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.862830    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8a8d1736-1ef0-428e-866b-3f8f796c4471-node-exporter-tls\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.869598 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.869543    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26qj\" (UniqueName: \"kubernetes.io/projected/8a8d1736-1ef0-428e-866b-3f8f796c4471-kube-api-access-r26qj\") pod \"node-exporter-l9qfx\" (UID: \"8a8d1736-1ef0-428e-866b-3f8f796c4471\") " pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:17.961248 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.961216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:17.973231 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:17.973208    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l9qfx"
Jul 19 16:43:18.003306 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:43:18.003259    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8d1736_1ef0_428e_866b_3f8f796c4471.slice/crio-e45b790ce3998af77a13a86c4e3bbf234ae694d8abf465069712152a2f33f3b2 WatchSource:0}: Error finding container e45b790ce3998af77a13a86c4e3bbf234ae694d8abf465069712152a2f33f3b2: Status 404 returned error can't find the container with id e45b790ce3998af77a13a86c4e3bbf234ae694d8abf465069712152a2f33f3b2
Jul 19 16:43:18.005124 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:18.005107    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 16:43:18.229319 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:18.229270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:18.299420 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:18.299375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:18.618772 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:18.618679    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9qfx" event={"ID":"8a8d1736-1ef0-428e-866b-3f8f796c4471","Type":"ContainerStarted","Data":"e45b790ce3998af77a13a86c4e3bbf234ae694d8abf465069712152a2f33f3b2"}
Jul 19 16:43:18.837634 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:18.837601    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:19.627358 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:19.627318    2584 generic.go:356] "Generic (PLEG): container finished" podID="8a8d1736-1ef0-428e-866b-3f8f796c4471" containerID="fa62bae2ca3843b96cd6a0b00254c19506ab4c6fcd45d1b53fb6d6dbd27fc111" exitCode=0
Jul 19 16:43:19.627758 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:19.627381    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9qfx" event={"ID":"8a8d1736-1ef0-428e-866b-3f8f796c4471","Type":"ContainerDied","Data":"fa62bae2ca3843b96cd6a0b00254c19506ab4c6fcd45d1b53fb6d6dbd27fc111"}
Jul 19 16:43:20.179497 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.179466    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:20.280456 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.280428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:20.476021 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.475926    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:20.632225 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.632191    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9qfx" event={"ID":"8a8d1736-1ef0-428e-866b-3f8f796c4471","Type":"ContainerStarted","Data":"38bce23a3e4b05a7ebfe095cec41ccb2e22fbf0a4f9a00c5585112e4b3f6ce19"}
Jul 19 16:43:20.632781 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.632250    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l9qfx" event={"ID":"8a8d1736-1ef0-428e-866b-3f8f796c4471","Type":"ContainerStarted","Data":"635e6ac8dccc749aff6e1f1c3c1256b6bdc401d4ea3cc98da1a2a2d728e6aa7c"}
Jul 19 16:43:20.654890 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.654842    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l9qfx" podStartSLOduration=2.541108856 podStartE2EDuration="3.654825313s" totalImagesPullingTime="1.041888953s" totalInitContainerRuntime="71.827504ms" isStatefulPod=true podCreationTimestamp="2026-07-19 16:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:43:20.653154317 +0000 UTC m=+310.558405740" watchObservedRunningTime="2026-07-19 16:43:20.654825313 +0000 UTC m=+310.560076736"
Jul 19 16:43:20.765109 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.765020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:20.824173 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.824138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:20.881900 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:20.881872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:21.549711 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.549673    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"]
Jul 19 16:43:21.551653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.551631    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.555123 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.555100    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h25vq\""
Jul 19 16:43:21.555332 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.555106    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 19 16:43:21.555332 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.555110    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 19 16:43:21.555508 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.555102    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 19 16:43:21.555508 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.555187    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-88ofj0fl1ducs\""
Jul 19 16:43:21.555508 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.555202    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 19 16:43:21.562983 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.562960    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"]
Jul 19 16:43:21.583883 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.583853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-secret-metrics-server-tls\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.584011 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.583887    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-client-ca-bundle\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.584011 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.583945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/932cc83c-0424-4040-a701-e456d4186fcc-metrics-server-audit-profiles\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.584011 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.583980    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-secret-metrics-server-client-certs\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.584131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.584017    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/932cc83c-0424-4040-a701-e456d4186fcc-audit-log\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.584131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.584048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932cc83c-0424-4040-a701-e456d4186fcc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.584131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.584087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dgmd\" (UniqueName: \"kubernetes.io/projected/932cc83c-0424-4040-a701-e456d4186fcc-kube-api-access-7dgmd\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.626988 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.626956    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:43:21.685155 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/932cc83c-0424-4040-a701-e456d4186fcc-metrics-server-audit-profiles\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-secret-metrics-server-client-certs\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685205    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/932cc83c-0424-4040-a701-e456d4186fcc-audit-log\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685244    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932cc83c-0424-4040-a701-e456d4186fcc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685272    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dgmd\" (UniqueName: \"kubernetes.io/projected/932cc83c-0424-4040-a701-e456d4186fcc-kube-api-access-7dgmd\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-secret-metrics-server-tls\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-client-ca-bundle\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.685786 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.685667    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/932cc83c-0424-4040-a701-e456d4186fcc-audit-log\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.686389 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.686365    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/932cc83c-0424-4040-a701-e456d4186fcc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.686500 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.686456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/932cc83c-0424-4040-a701-e456d4186fcc-metrics-server-audit-profiles\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.688047 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.688011    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-secret-metrics-server-client-certs\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.688047 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.688041    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-client-ca-bundle\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.688336 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.688317    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/932cc83c-0424-4040-a701-e456d4186fcc-secret-metrics-server-tls\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.693514 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.693495    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dgmd\" (UniqueName: \"kubernetes.io/projected/932cc83c-0424-4040-a701-e456d4186fcc-kube-api-access-7dgmd\") pod \"metrics-server-7ddfd74fc-4nnhk\" (UID: \"932cc83c-0424-4040-a701-e456d4186fcc\") " pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.860928 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.860831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:21.989180 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:21.989152    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"]
Jul 19 16:43:21.992209 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:43:21.992180    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932cc83c_0424_4040_a701_e456d4186fcc.slice/crio-12e0bfb9db5e3b4684ddea79e3788a8aa853497e235de842df840a0595523cdb WatchSource:0}: Error finding container 12e0bfb9db5e3b4684ddea79e3788a8aa853497e235de842df840a0595523cdb: Status 404 returned error can't find the container with id 12e0bfb9db5e3b4684ddea79e3788a8aa853497e235de842df840a0595523cdb
Jul 19 16:43:22.034567 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:22.034537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:22.093985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:22.093943    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:22.234190 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:22.234160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:22.640468 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:22.640379    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk" event={"ID":"932cc83c-0424-4040-a701-e456d4186fcc","Type":"ContainerStarted","Data":"12e0bfb9db5e3b4684ddea79e3788a8aa853497e235de842df840a0595523cdb"}
Jul 19 16:43:23.437450 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.437413    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:23.625993 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.625962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:23.646354 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.646320    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk" event={"ID":"932cc83c-0424-4040-a701-e456d4186fcc","Type":"ContainerStarted","Data":"688c34e2521fc58a302523d435076e7dc1457725a632b8a9f623fdcd23a68a57"}
Jul 19 16:43:23.662007 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.661963    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk" podStartSLOduration=1.123417284 podStartE2EDuration="2.661948942s" totalImagesPullingTime="1.538531658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:43:23.661506972 +0000 UTC m=+313.566758396" watchObservedRunningTime="2026-07-19 16:43:23.661948942 +0000 UTC m=+313.567200364"
Jul 19 16:43:23.729045 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.728957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:23.829575 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.829543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:23.889590 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:23.889566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:26.122546 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:26.122514    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:26.172679 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:26.172647    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:33.678003 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:33.677970    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:34.045311 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:34.045213    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:34.151517 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:34.151478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:34.247756 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:34.247724    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:34.303236 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:34.303134    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:35.198729 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:35.198682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:35.304741 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:35.304704    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:38.054832 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:38.054798    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:38.104956 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:38.104926    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:41.861270 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:41.861235    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:41.861672 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:41.861316    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:43:53.064400 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:53.064368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:43:53.116519 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:43:53.116480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:01.867533 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:01.867450    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:44:01.871229 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:01.871202    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7ddfd74fc-4nnhk"
Jul 19 16:44:05.964811 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:05.964771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:20.180974 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:20.180940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:20.233772 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:20.233739    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:24.484022 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:24.483988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:44:24.486441 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:24.486420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c09c151-0fa7-4b87-98b9-7b5cf9aedeff-metrics-certs\") pod \"network-metrics-daemon-xqq8r\" (UID: \"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff\") " pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:44:24.629873 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:24.629841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pkhnw\""
Jul 19 16:44:24.637857 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:24.637839    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xqq8r"
Jul 19 16:44:24.765329 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:24.765249    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xqq8r"]
Jul 19 16:44:24.767791 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:44:24.767761    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c09c151_0fa7_4b87_98b9_7b5cf9aedeff.slice/crio-9fcd305a2f0958ef4ecd000313b6215a824d4d9e35d7e1ef2853ea823e903439 WatchSource:0}: Error finding container 9fcd305a2f0958ef4ecd000313b6215a824d4d9e35d7e1ef2853ea823e903439: Status 404 returned error can't find the container with id 9fcd305a2f0958ef4ecd000313b6215a824d4d9e35d7e1ef2853ea823e903439
Jul 19 16:44:24.841547 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:24.841517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xqq8r" event={"ID":"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff","Type":"ContainerStarted","Data":"9fcd305a2f0958ef4ecd000313b6215a824d4d9e35d7e1ef2853ea823e903439"}
Jul 19 16:44:25.848162 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:25.848128    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xqq8r" event={"ID":"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff","Type":"ContainerStarted","Data":"0faefa27114945b26b84271e8c3fb04100f1666c5583334ef5dc15dc44b4eae2"}
Jul 19 16:44:25.848571 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:25.848189    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xqq8r" event={"ID":"3c09c151-0fa7-4b87-98b9-7b5cf9aedeff","Type":"ContainerStarted","Data":"da2cb6e31c3da9ab90b02dc001aa573e2e9d31be961100d680a3afa4543cb153"}
Jul 19 16:44:25.862691 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:25.862637    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xqq8r" podStartSLOduration=374.969263585 podStartE2EDuration="6m15.862621145s" totalImagesPullingTime="893.35756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:44:25.862254238 +0000 UTC m=+375.767505663" watchObservedRunningTime="2026-07-19 16:44:25.862621145 +0000 UTC m=+375.767872569"
Jul 19 16:44:25.945760 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:25.945681    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:26.001934 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:26.001909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:36.437831 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:36.437789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:36.822728 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:36.822644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:41.012305 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:41.012243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:41.071687 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:41.071653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:44.973447 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:44.973417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:45.031168 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:45.031132    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:56.437861 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:56.437830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:44:56.537004 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:44:56.536966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:45:51.662985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:45:51.662938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:45:51.715212 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:45:51.715183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:45:52.607354 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:45:52.607324    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:45:53.856801 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:45:53.856770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:45:53.907444 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:45:53.907409    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:03.532930 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:03.532872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:03.585717 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:03.585690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:04.665176 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:04.665142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:04.716298 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:04.716253    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:08.084926 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:08.084895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:08.185147 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:08.185118    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:08.738163 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:08.738127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:15.158518 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:15.158480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:53.277634 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:53.277599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:46:53.374411 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:46:53.374378    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:14.862532 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:14.862438    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:15.660191 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:15.660151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:18.457053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:18.457014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:18.557695 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:18.557659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:19.245361 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:19.245329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:19.345229 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:19.345198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:20.451339 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:20.451302    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:20.529383 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:20.529353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:22.471723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:22.471689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:22.519234 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:22.519193    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:22.849521 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:22.849437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:25.262541 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:25.262509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:25.363681 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:25.363651    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:29.805685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:29.805652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:29.960738 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:29.960702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:31.066339 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:31.066305    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:37.154899 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:37.154863    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:37.209713 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:37.209682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:42.568099 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:42.568066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:42.668831 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:42.668796    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:44.233596 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.233564    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"]
Jul 19 16:47:44.236510 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.236494    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.238988 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.238965    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 19 16:47:44.239103 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.238967    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 19 16:47:44.239153 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.239104    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wckp8\""
Jul 19 16:47:44.244848 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.244826    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"]
Jul 19 16:47:44.375363 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.375324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7gw\" (UniqueName: \"kubernetes.io/projected/c71c323d-89bf-4708-bf38-95f2cd40efa9-kube-api-access-8z7gw\") pod \"openshift-lws-operator-fd8ccff4c-fmjbt\" (UID: \"c71c323d-89bf-4708-bf38-95f2cd40efa9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.375543 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.375386    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c71c323d-89bf-4708-bf38-95f2cd40efa9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmjbt\" (UID: \"c71c323d-89bf-4708-bf38-95f2cd40efa9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.476122 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.476087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7gw\" (UniqueName: \"kubernetes.io/projected/c71c323d-89bf-4708-bf38-95f2cd40efa9-kube-api-access-8z7gw\") pod \"openshift-lws-operator-fd8ccff4c-fmjbt\" (UID: \"c71c323d-89bf-4708-bf38-95f2cd40efa9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.476326 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.476139    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c71c323d-89bf-4708-bf38-95f2cd40efa9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmjbt\" (UID: \"c71c323d-89bf-4708-bf38-95f2cd40efa9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.476501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.476484    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c71c323d-89bf-4708-bf38-95f2cd40efa9-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fmjbt\" (UID: \"c71c323d-89bf-4708-bf38-95f2cd40efa9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.486993 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.486925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7gw\" (UniqueName: \"kubernetes.io/projected/c71c323d-89bf-4708-bf38-95f2cd40efa9-kube-api-access-8z7gw\") pod \"openshift-lws-operator-fd8ccff4c-fmjbt\" (UID: \"c71c323d-89bf-4708-bf38-95f2cd40efa9\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.545927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.545890    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"
Jul 19 16:47:44.678426 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:44.678396    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt"]
Jul 19 16:47:44.681326 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:47:44.681298    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc71c323d_89bf_4708_bf38_95f2cd40efa9.slice/crio-4af697cd660bcba40296e1352cf7bed1bd5a743a10a233fa302c37775e5b8657 WatchSource:0}: Error finding container 4af697cd660bcba40296e1352cf7bed1bd5a743a10a233fa302c37775e5b8657: Status 404 returned error can't find the container with id 4af697cd660bcba40296e1352cf7bed1bd5a743a10a233fa302c37775e5b8657
Jul 19 16:47:45.495756 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:45.495713    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt" event={"ID":"c71c323d-89bf-4708-bf38-95f2cd40efa9","Type":"ContainerStarted","Data":"4af697cd660bcba40296e1352cf7bed1bd5a743a10a233fa302c37775e5b8657"}
Jul 19 16:47:47.504421 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:47.504333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt" event={"ID":"c71c323d-89bf-4708-bf38-95f2cd40efa9","Type":"ContainerStarted","Data":"b46edbcb589da7395161afa0a0966b0ff6648d184fce847e09f4054daab92687"}
Jul 19 16:47:47.521108 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:47.521062    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fmjbt" podStartSLOduration=1.041003795 podStartE2EDuration="3.521045954s" totalImagesPullingTime="2.480042159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:47:47.52010859 +0000 UTC m=+577.425360014" watchObservedRunningTime="2026-07-19 16:47:47.521045954 +0000 UTC m=+577.426297378"
Jul 19 16:47:48.170907 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:48.170877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:48.271615 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:48.271580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:47:49.884055 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:47:49.884008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:00.097004 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:00.096971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:00.197528 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:00.197490    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:03.940871 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:03.940832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:04.040536 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.040504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:04.579502 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.579378    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:04.794008 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.793975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:04.900676 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.900644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:04.924154 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.924125    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"]
Jul 19 16:48:04.927104 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.927088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:04.930096 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.930074    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 19 16:48:04.930232 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.930127    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 19 16:48:04.930232 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.930202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9w89c\""
Jul 19 16:48:04.930386 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.930373    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 19 16:48:04.930796 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.930776    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 19 16:48:04.947860 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:04.947829    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"]
Jul 19 16:48:05.024804 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.024765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcdf\" (UniqueName: \"kubernetes.io/projected/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-kube-api-access-gtcdf\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.024804 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.024802    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-manifests\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.025012 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.024826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-tmp\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.025012 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.024898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.025012 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.024945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.064189 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.064152    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:05.125262 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.125229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.125425 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.125305    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.125425 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.125337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcdf\" (UniqueName: \"kubernetes.io/projected/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-kube-api-access-gtcdf\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.125425 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.125356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-manifests\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.125425 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.125380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-tmp\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.125794 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.125770    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-manifests\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.127747 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.127725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-tmp\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.127995 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.127978    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.128127 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.128107    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.136505 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.136476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcdf\" (UniqueName: \"kubernetes.io/projected/2d5007b6-c0f3-40c1-bfb3-7cd5af81168b-kube-api-access-gtcdf\") pod \"opendatahub-operator-controller-manager-5f7db45576-9l62l\" (UID: \"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b\") " pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.161178 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.161097    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:05.236942 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.236916    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:05.267618 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.267587    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:05.364971 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.364940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:05.370835 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.370813    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"]
Jul 19 16:48:05.373393 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:05.373357    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5007b6_c0f3_40c1_bfb3_7cd5af81168b.slice/crio-a04b6bd6af98d081788ff1c37576e3c83a65e7dcb596bd6b9a6394ce2792c8f8 WatchSource:0}: Error finding container a04b6bd6af98d081788ff1c37576e3c83a65e7dcb596bd6b9a6394ce2792c8f8: Status 404 returned error can't find the container with id a04b6bd6af98d081788ff1c37576e3c83a65e7dcb596bd6b9a6394ce2792c8f8
Jul 19 16:48:05.464922 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.464834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:05.567779 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:05.567747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l" event={"ID":"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b","Type":"ContainerStarted","Data":"a04b6bd6af98d081788ff1c37576e3c83a65e7dcb596bd6b9a6394ce2792c8f8"}
Jul 19 16:48:08.582131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:08.582038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l" event={"ID":"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b","Type":"ContainerStarted","Data":"f43af966534c9936c6a2ac9f83c63e9f08850022e1adab99911d37265174ae49"}
Jul 19 16:48:09.587123 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:09.587084    2584 generic.go:356] "Generic (PLEG): container finished" podID="2d5007b6-c0f3-40c1-bfb3-7cd5af81168b" containerID="f43af966534c9936c6a2ac9f83c63e9f08850022e1adab99911d37265174ae49" exitCode=0
Jul 19 16:48:09.587623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:09.587143    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l" event={"ID":"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b","Type":"ContainerDied","Data":"f43af966534c9936c6a2ac9f83c63e9f08850022e1adab99911d37265174ae49"}
Jul 19 16:48:10.548934 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:10.548908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:48:10.550453 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:10.550427    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:48:10.593318 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:10.593258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l" event={"ID":"2d5007b6-c0f3-40c1-bfb3-7cd5af81168b","Type":"ContainerStarted","Data":"51226e1284bf5d821a6b8e3491824a834cf852c8ec864cfc3777f5925d0db63b"}
Jul 19 16:48:10.593703 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:10.593363    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:10.619232 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:10.614818    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l" podStartSLOduration=2.343871967 podStartE2EDuration="6.614797439s" totalImagesPullingTime="3.256359189s" totalInitContainerRuntime="1.014566283s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:10.613423795 +0000 UTC m=+600.518675220" watchObservedRunningTime="2026-07-19 16:48:10.614797439 +0000 UTC m=+600.520048867"
Jul 19 16:48:16.794577 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:16.794538    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:17.992213 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:17.992172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:21.598949 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:21.598920    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f7db45576-9l62l"
Jul 19 16:48:23.768995 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:23.768957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:23.903835 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:23.903793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:24.280494 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.280450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:24.360808 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.360771    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"]
Jul 19 16:48:24.369364 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.369343    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.371794 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.371773    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xhhgc\""
Jul 19 16:48:24.371794 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.371790    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 19 16:48:24.372783 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.372768    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 19 16:48:24.372835 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.372784    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 19 16:48:24.372872 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.372772    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 19 16:48:24.376399 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.376376    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"]
Jul 19 16:48:24.471006 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.470971    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-tls-certs\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.471214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.471069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v4p7\" (UniqueName: \"kubernetes.io/projected/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-kube-api-access-7v4p7\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.471214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.471133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-tmp\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.471214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.471156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:24.505108 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.505070    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nvpc8"]
Jul 19 16:48:24.507137 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.507116    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:24.511296 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.511261    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 19 16:48:24.511449 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.511428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-m6bmn\""
Jul 19 16:48:24.518716 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.518692    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nvpc8"]
Jul 19 16:48:24.560493 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.560405    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:24.572672 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.572637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-tmp\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.572847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.572694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314008-516a-4d6b-a1b6-997f61bdfa44-cert\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:24.572847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.572729    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-tls-certs\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.572847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.572767    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4qzt\" (UniqueName: \"kubernetes.io/projected/49314008-516a-4d6b-a1b6-997f61bdfa44-kube-api-access-b4qzt\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:24.572847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.572811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v4p7\" (UniqueName: \"kubernetes.io/projected/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-kube-api-access-7v4p7\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.575143 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.575117    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-tmp\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.575402 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.575386    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-tls-certs\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.584673 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.584640    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v4p7\" (UniqueName: \"kubernetes.io/projected/dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2-kube-api-access-7v4p7\") pod \"kube-auth-proxy-cdf6466d9-t8sbq\" (UID: \"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2\") " pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.661482 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.661448    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:24.673560 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.673536    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314008-516a-4d6b-a1b6-997f61bdfa44-cert\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:24.673645 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.673579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4qzt\" (UniqueName: \"kubernetes.io/projected/49314008-516a-4d6b-a1b6-997f61bdfa44-kube-api-access-b4qzt\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:24.673702 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:48:24.673678    2584 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 19 16:48:24.673753 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:48:24.673743    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49314008-516a-4d6b-a1b6-997f61bdfa44-cert podName:49314008-516a-4d6b-a1b6-997f61bdfa44 nodeName:}" failed. No retries permitted until 2026-07-19 16:48:25.173726425 +0000 UTC m=+615.078977827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/49314008-516a-4d6b-a1b6-997f61bdfa44-cert") pod "odh-model-controller-7998c67dc4-nvpc8" (UID: "49314008-516a-4d6b-a1b6-997f61bdfa44") : secret "odh-model-controller-webhook-cert" not found
Jul 19 16:48:24.680468 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.680436    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"
Jul 19 16:48:24.693105 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.693064    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4qzt\" (UniqueName: \"kubernetes.io/projected/49314008-516a-4d6b-a1b6-997f61bdfa44-kube-api-access-b4qzt\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:24.828896 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.828872    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq"]
Jul 19 16:48:24.831342 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:24.831316    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfea2f8_69f5_4901_8eab_ac3b9cf5c4e2.slice/crio-01491be5140fc0d0ec277900468599d6c521d2f3bce932d571312538e9a728e2 WatchSource:0}: Error finding container 01491be5140fc0d0ec277900468599d6c521d2f3bce932d571312538e9a728e2: Status 404 returned error can't find the container with id 01491be5140fc0d0ec277900468599d6c521d2f3bce932d571312538e9a728e2
Jul 19 16:48:24.833693 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:24.833675    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 16:48:25.110732 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.110634    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:25.178841 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.178803    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314008-516a-4d6b-a1b6-997f61bdfa44-cert\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:25.181401 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.181376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49314008-516a-4d6b-a1b6-997f61bdfa44-cert\") pod \"odh-model-controller-7998c67dc4-nvpc8\" (UID: \"49314008-516a-4d6b-a1b6-997f61bdfa44\") " pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:25.203793 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.203759    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:25.230605 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.230570    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7mcg4"]
Jul 19 16:48:25.233094 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.233078    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.236727 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.236705    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-bgpqb\""
Jul 19 16:48:25.236834 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.236705    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 19 16:48:25.248662 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.248635    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7mcg4"]
Jul 19 16:48:25.279668 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.279635    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzkb\" (UniqueName: \"kubernetes.io/projected/65c44697-b060-45ad-8346-c502b291e9ac-kube-api-access-qnzkb\") pod \"model-serving-api-54d89bd5fc-7mcg4\" (UID: \"65c44697-b060-45ad-8346-c502b291e9ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.279839 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.279684    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c44697-b060-45ad-8346-c502b291e9ac-tls-certs\") pod \"model-serving-api-54d89bd5fc-7mcg4\" (UID: \"65c44697-b060-45ad-8346-c502b291e9ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.364323 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.364207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:25.380421 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.380388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzkb\" (UniqueName: \"kubernetes.io/projected/65c44697-b060-45ad-8346-c502b291e9ac-kube-api-access-qnzkb\") pod \"model-serving-api-54d89bd5fc-7mcg4\" (UID: \"65c44697-b060-45ad-8346-c502b291e9ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.380533 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.380436    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c44697-b060-45ad-8346-c502b291e9ac-tls-certs\") pod \"model-serving-api-54d89bd5fc-7mcg4\" (UID: \"65c44697-b060-45ad-8346-c502b291e9ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.382983 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.382956    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65c44697-b060-45ad-8346-c502b291e9ac-tls-certs\") pod \"model-serving-api-54d89bd5fc-7mcg4\" (UID: \"65c44697-b060-45ad-8346-c502b291e9ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.388535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.388503    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzkb\" (UniqueName: \"kubernetes.io/projected/65c44697-b060-45ad-8346-c502b291e9ac-kube-api-access-qnzkb\") pod \"model-serving-api-54d89bd5fc-7mcg4\" (UID: \"65c44697-b060-45ad-8346-c502b291e9ac\") " pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.418721 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.418690    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:25.447506 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.447438    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f948b"]
Jul 19 16:48:25.449826 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.449782    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.451662 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.451634    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:25.452376 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.452354    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 19 16:48:25.452557 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.452533    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 19 16:48:25.452892 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.452863    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 19 16:48:25.452892 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.452884    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m45md\""
Jul 19 16:48:25.453075 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.452867    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 19 16:48:25.464850 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.464813    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f948b"]
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/449756a7-c36e-4578-8559-b00a5393fb67-local-certs\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486567    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486664    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486706    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-istio-token\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-cacerts\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.495132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.486818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cds9t\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-kube-api-access-cds9t\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.505597 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.505474    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:25.543410 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.543372    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:25.586578 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.586548    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-nvpc8"]
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cds9t\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-kube-api-access-cds9t\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588195    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588233    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/449756a7-c36e-4578-8559-b00a5393fb67-local-certs\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588271    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588327    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588362    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-istio-token\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.589146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.588411    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-cacerts\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.591393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.591339    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.594571 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:25.594536    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49314008_516a_4d6b_a1b6_997f61bdfa44.slice/crio-b4ab1fc79f5ff30669a6f9e0f894f8426481b460607e79f32cfdca9be4c3ff01 WatchSource:0}: Error finding container b4ab1fc79f5ff30669a6f9e0f894f8426481b460607e79f32cfdca9be4c3ff01: Status 404 returned error can't find the container with id b4ab1fc79f5ff30669a6f9e0f894f8426481b460607e79f32cfdca9be4c3ff01
Jul 19 16:48:25.598811 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.598779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-istio-token\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.599814 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.599768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cds9t\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-kube-api-access-cds9t\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.601119 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.600842    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.601119 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.601074    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-cacerts\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.602794 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.602589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.602794 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.602722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/449756a7-c36e-4578-8559-b00a5393fb67-local-certs\") pod \"istiod-openshift-gateway-7574459656-f948b\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.656816 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.656776    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8" event={"ID":"49314008-516a-4d6b-a1b6-997f61bdfa44","Type":"ContainerStarted","Data":"b4ab1fc79f5ff30669a6f9e0f894f8426481b460607e79f32cfdca9be4c3ff01"}
Jul 19 16:48:25.658208 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.658178    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq" event={"ID":"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2","Type":"ContainerStarted","Data":"01491be5140fc0d0ec277900468599d6c521d2f3bce932d571312538e9a728e2"}
Jul 19 16:48:25.707798 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.707594    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7mcg4"]
Jul 19 16:48:25.710779 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:25.710743    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c44697_b060_45ad_8346_c502b291e9ac.slice/crio-b7c93db5427fe5a8440cf9ed650e304858839236b4d3a2b309b76104ee2d3062 WatchSource:0}: Error finding container b7c93db5427fe5a8440cf9ed650e304858839236b4d3a2b309b76104ee2d3062: Status 404 returned error can't find the container with id b7c93db5427fe5a8440cf9ed650e304858839236b4d3a2b309b76104ee2d3062
Jul 19 16:48:25.762588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.762552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:25.945590 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:25.945547    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f948b"]
Jul 19 16:48:25.948561 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:25.948520    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449756a7_c36e_4578_8559_b00a5393fb67.slice/crio-036eb42b06d31aa1ced6c885afdf8212c90cf03ae7e05634a3fc1522fbcda819 WatchSource:0}: Error finding container 036eb42b06d31aa1ced6c885afdf8212c90cf03ae7e05634a3fc1522fbcda819: Status 404 returned error can't find the container with id 036eb42b06d31aa1ced6c885afdf8212c90cf03ae7e05634a3fc1522fbcda819
Jul 19 16:48:26.381650 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:26.381513    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:26.665245 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:26.665166    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" event={"ID":"449756a7-c36e-4578-8559-b00a5393fb67","Type":"ContainerStarted","Data":"036eb42b06d31aa1ced6c885afdf8212c90cf03ae7e05634a3fc1522fbcda819"}
Jul 19 16:48:26.666985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:26.666921    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4" event={"ID":"65c44697-b060-45ad-8346-c502b291e9ac","Type":"ContainerStarted","Data":"b7c93db5427fe5a8440cf9ed650e304858839236b4d3a2b309b76104ee2d3062"}
Jul 19 16:48:30.726979 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:30.726939    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 19 16:48:30.727390 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:30.727036    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 19 16:48:31.616900 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.616809    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:31.692020 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.691973    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4" event={"ID":"65c44697-b060-45ad-8346-c502b291e9ac","Type":"ContainerStarted","Data":"447a9ceb245e7c5751826962bbb16d2979ae3b4036167b14242978f20efcc44b"}
Jul 19 16:48:31.693038 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.693011    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:48:31.694969 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.694936    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8" event={"ID":"49314008-516a-4d6b-a1b6-997f61bdfa44","Type":"ContainerStarted","Data":"2cbd09d4462caa3700880e1ac966a774cac46937466c8291e39de8ec7a48cf2e"}
Jul 19 16:48:31.695105 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.695086    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:31.696576 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.696553    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq" event={"ID":"dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2","Type":"ContainerStarted","Data":"2aa36fa7a673df8091ea754ebe412c3fed873d00d050901a1d0beeff4e78aa98"}
Jul 19 16:48:31.697961 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.697943    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" event={"ID":"449756a7-c36e-4578-8559-b00a5393fb67","Type":"ContainerStarted","Data":"3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759"}
Jul 19 16:48:31.698141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.698126    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:31.699876 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.699855    2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-f948b container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 19 16:48:31.699953 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.699899    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" podUID="449756a7-c36e-4578-8559-b00a5393fb67" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 19 16:48:31.711495 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.711456    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4" podStartSLOduration=1.7148354970000002 podStartE2EDuration="6.711442246s" totalImagesPullingTime="4.996606749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:31.710244452 +0000 UTC m=+621.615495878" watchObservedRunningTime="2026-07-19 16:48:31.711442246 +0000 UTC m=+621.616693647"
Jul 19 16:48:31.720994 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.720968    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:31.733183 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.733136    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cdf6466d9-t8sbq" podStartSLOduration=1.911761828 podStartE2EDuration="7.733119284s" totalImagesPullingTime="5.821357456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:31.732008586 +0000 UTC m=+621.637260011" watchObservedRunningTime="2026-07-19 16:48:31.733119284 +0000 UTC m=+621.638370707"
Jul 19 16:48:31.747755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.747706    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8" podStartSLOduration=2.626603994 podStartE2EDuration="7.74769174s" totalImagesPullingTime="5.121087746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:31.746490948 +0000 UTC m=+621.651742374" watchObservedRunningTime="2026-07-19 16:48:31.74769174 +0000 UTC m=+621.652943163"
Jul 19 16:48:31.766984 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:31.766931    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" podStartSLOduration=1.991452952 podStartE2EDuration="6.7669131s" totalImagesPullingTime="4.775460148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:31.765038442 +0000 UTC m=+621.670289868" watchObservedRunningTime="2026-07-19 16:48:31.7669131 +0000 UTC m=+621.672164523"
Jul 19 16:48:32.304485 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:32.304452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:32.473498 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:32.473468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:32.596065 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:32.595981    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:32.705712 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:32.705688    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:35.232730 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.232693    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:35.346297 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.346244    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"]
Jul 19 16:48:35.349254 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.349225    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.350239 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.350216    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"]
Jul 19 16:48:35.351918 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.351890    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 19 16:48:35.352052 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.352033    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-5d82b\""
Jul 19 16:48:35.353245 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.353232    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:35.355819 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.355799    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6xcsv\""
Jul 19 16:48:35.356042 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.356025    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 19 16:48:35.360586 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.360564    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"]
Jul 19 16:48:35.369983 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.369959    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"]
Jul 19 16:48:35.388407 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.388376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:35.507487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.507402    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq9vs\" (UniqueName: \"kubernetes.io/projected/3ebd0261-2822-4192-9050-03e2f99975c8-kube-api-access-rq9vs\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:35.507487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.507462    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/542eaa8b-5b41-431b-98a3-f47182937dba-cert\") pod \"kserve-controller-manager-d6f4b65b8-txgvt\" (UID: \"542eaa8b-5b41-431b-98a3-f47182937dba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.507652 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.507515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln796\" (UniqueName: \"kubernetes.io/projected/542eaa8b-5b41-431b-98a3-f47182937dba-kube-api-access-ln796\") pod \"kserve-controller-manager-d6f4b65b8-txgvt\" (UID: \"542eaa8b-5b41-431b-98a3-f47182937dba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.507652 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.507547    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ebd0261-2822-4192-9050-03e2f99975c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:35.551844 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.551814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:35.608166 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.608136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ebd0261-2822-4192-9050-03e2f99975c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:35.608375 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.608188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq9vs\" (UniqueName: \"kubernetes.io/projected/3ebd0261-2822-4192-9050-03e2f99975c8-kube-api-access-rq9vs\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:35.608375 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.608231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/542eaa8b-5b41-431b-98a3-f47182937dba-cert\") pod \"kserve-controller-manager-d6f4b65b8-txgvt\" (UID: \"542eaa8b-5b41-431b-98a3-f47182937dba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.608375 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.608258    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln796\" (UniqueName: \"kubernetes.io/projected/542eaa8b-5b41-431b-98a3-f47182937dba-kube-api-access-ln796\") pod \"kserve-controller-manager-d6f4b65b8-txgvt\" (UID: \"542eaa8b-5b41-431b-98a3-f47182937dba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.608375 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:48:35.608312    2584 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 19 16:48:35.608536 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:48:35.608404    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ebd0261-2822-4192-9050-03e2f99975c8-cert podName:3ebd0261-2822-4192-9050-03e2f99975c8 nodeName:}" failed. No retries permitted until 2026-07-19 16:48:36.108386313 +0000 UTC m=+626.013637715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3ebd0261-2822-4192-9050-03e2f99975c8-cert") pod "llmisvc-controller-manager-777c6bd7d5-2hmq5" (UID: "3ebd0261-2822-4192-9050-03e2f99975c8") : secret "llmisvc-webhook-server-cert" not found
Jul 19 16:48:35.610970 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.610950    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/542eaa8b-5b41-431b-98a3-f47182937dba-cert\") pod \"kserve-controller-manager-d6f4b65b8-txgvt\" (UID: \"542eaa8b-5b41-431b-98a3-f47182937dba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.619561 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.619536    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln796\" (UniqueName: \"kubernetes.io/projected/542eaa8b-5b41-431b-98a3-f47182937dba-kube-api-access-ln796\") pod \"kserve-controller-manager-d6f4b65b8-txgvt\" (UID: \"542eaa8b-5b41-431b-98a3-f47182937dba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.623434 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.623414    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq9vs\" (UniqueName: \"kubernetes.io/projected/3ebd0261-2822-4192-9050-03e2f99975c8-kube-api-access-rq9vs\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:35.662535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.662500    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:35.793799 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:35.793707    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"]
Jul 19 16:48:35.796437 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:35.796411    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542eaa8b_5b41_431b_98a3_f47182937dba.slice/crio-cde5eb3d3896ca46b9d1cb05d1e1ee8eef111a6abb6af27db96ff99eed231d14 WatchSource:0}: Error finding container cde5eb3d3896ca46b9d1cb05d1e1ee8eef111a6abb6af27db96ff99eed231d14: Status 404 returned error can't find the container with id cde5eb3d3896ca46b9d1cb05d1e1ee8eef111a6abb6af27db96ff99eed231d14
Jul 19 16:48:36.003491 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.003460    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:36.113898 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.113813    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ebd0261-2822-4192-9050-03e2f99975c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:36.116367 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.116346    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ebd0261-2822-4192-9050-03e2f99975c8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2hmq5\" (UID: \"3ebd0261-2822-4192-9050-03e2f99975c8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:36.268601 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.268562    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:36.411685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.411659    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"]
Jul 19 16:48:36.414834 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:48:36.414804    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3ebd0261_2822_4192_9050_03e2f99975c8.slice/crio-3dc765126099064f0ecaadd4ca5c2175a63234d51f98292433677d2078a01908 WatchSource:0}: Error finding container 3dc765126099064f0ecaadd4ca5c2175a63234d51f98292433677d2078a01908: Status 404 returned error can't find the container with id 3dc765126099064f0ecaadd4ca5c2175a63234d51f98292433677d2078a01908
Jul 19 16:48:36.551081 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.551048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:36.653523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.653490    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:36.723272 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.723168    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt" event={"ID":"542eaa8b-5b41-431b-98a3-f47182937dba","Type":"ContainerStarted","Data":"cde5eb3d3896ca46b9d1cb05d1e1ee8eef111a6abb6af27db96ff99eed231d14"}
Jul 19 16:48:36.724214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.724193    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5" event={"ID":"3ebd0261-2822-4192-9050-03e2f99975c8","Type":"ContainerStarted","Data":"3dc765126099064f0ecaadd4ca5c2175a63234d51f98292433677d2078a01908"}
Jul 19 16:48:36.755266 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:36.755214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:48:38.736622 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:38.736581    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt" event={"ID":"542eaa8b-5b41-431b-98a3-f47182937dba","Type":"ContainerStarted","Data":"2def320592627fff8802456a1a20c3683fbfbfe99d674ba3ec9298f3bfad26db"}
Jul 19 16:48:38.737032 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:38.736750    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:48:38.755816 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:38.755447    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt" podStartSLOduration=1.24187566 podStartE2EDuration="3.755427015s" totalImagesPullingTime="2.513551355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:38.752355225 +0000 UTC m=+628.657606649" watchObservedRunningTime="2026-07-19 16:48:38.755427015 +0000 UTC m=+628.660678441"
Jul 19 16:48:39.742811 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:39.742776    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5" event={"ID":"3ebd0261-2822-4192-9050-03e2f99975c8","Type":"ContainerStarted","Data":"fc5f6af3938493a60209fa2777b7148dc14213d41ee28efa7a635ff4c973ee12"}
Jul 19 16:48:39.759571 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:39.759511    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5" podStartSLOduration=2.127303297 podStartE2EDuration="4.759493833s" totalImagesPullingTime="2.632190536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:48:39.758386495 +0000 UTC m=+629.663637934" watchObservedRunningTime="2026-07-19 16:48:39.759493833 +0000 UTC m=+629.664745258"
Jul 19 16:48:39.950564 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:39.950528    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f948b"]
Jul 19 16:48:39.950781 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:39.950754    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" podUID="449756a7-c36e-4578-8559-b00a5393fb67" containerName="discovery" containerID="cri-o://3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759" gracePeriod=30
Jul 19 16:48:40.195741 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.195716    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:40.352343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352228    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/449756a7-c36e-4578-8559-b00a5393fb67-local-certs\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352266    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-dns-cert\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352334    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-kubeconfig\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352358    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-cacerts\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352395    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-istio-token\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352458    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-ca-configmap\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cds9t\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-kube-api-access-cds9t\") pod \"449756a7-c36e-4578-8559-b00a5393fb67\" (UID: \"449756a7-c36e-4578-8559-b00a5393fb67\") "
Jul 19 16:48:40.352920 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.352891    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 16:48:40.353091 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.353008    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-ca-configmap\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.354940 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.354891    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-cacerts" (OuterVolumeSpecName: "cacerts") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:48:40.354940 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.354898    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-kube-api-access-cds9t" (OuterVolumeSpecName: "kube-api-access-cds9t") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "kube-api-access-cds9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:48:40.355160 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.355139    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:48:40.355245 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.355213    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:48:40.355386 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.355274    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/449756a7-c36e-4578-8559-b00a5393fb67-local-certs" (OuterVolumeSpecName: "local-certs") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 16:48:40.355386 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.355338    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-istio-token" (OuterVolumeSpecName: "istio-token") pod "449756a7-c36e-4578-8559-b00a5393fb67" (UID: "449756a7-c36e-4578-8559-b00a5393fb67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:48:40.454240 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.454197    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cds9t\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-kube-api-access-cds9t\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.454240 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.454231    2584 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/449756a7-c36e-4578-8559-b00a5393fb67-local-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.454240 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.454241    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-csr-dns-cert\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.454240 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.454250    2584 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-istio-kubeconfig\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.454534 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.454260    2584 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/449756a7-c36e-4578-8559-b00a5393fb67-cacerts\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.454534 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.454267    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/449756a7-c36e-4578-8559-b00a5393fb67-istio-token\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:48:40.748375 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.748343    2584 generic.go:356] "Generic (PLEG): container finished" podID="449756a7-c36e-4578-8559-b00a5393fb67" containerID="3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759" exitCode=0
Jul 19 16:48:40.748936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.748392    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b"
Jul 19 16:48:40.748936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.748426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" event={"ID":"449756a7-c36e-4578-8559-b00a5393fb67","Type":"ContainerDied","Data":"3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759"}
Jul 19 16:48:40.748936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.748489    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-f948b" event={"ID":"449756a7-c36e-4578-8559-b00a5393fb67","Type":"ContainerDied","Data":"036eb42b06d31aa1ced6c885afdf8212c90cf03ae7e05634a3fc1522fbcda819"}
Jul 19 16:48:40.748936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.748519    2584 scope.go:117] "RemoveContainer" containerID="3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759"
Jul 19 16:48:40.748936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.748809    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:48:40.757301 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.757267    2584 scope.go:117] "RemoveContainer" containerID="3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759"
Jul 19 16:48:40.757617 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:48:40.757585    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759\": container with ID starting with 3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759 not found: ID does not exist" containerID="3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759"
Jul 19 16:48:40.757685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.757618    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759"} err="failed to get container status \"3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759\": rpc error: code = NotFound desc = could not find container \"3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759\": container with ID starting with 3145aaca9c6c479b3a15cfb4dac728221675ed0bb058b23c91af88fb4e2ae759 not found: ID does not exist"
Jul 19 16:48:40.766665 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.766638    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f948b"]
Jul 19 16:48:40.770860 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:40.770838    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-f948b"]
Jul 19 16:48:42.632136 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:42.632101    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449756a7-c36e-4578-8559-b00a5393fb67" path="/var/lib/kubelet/pods/449756a7-c36e-4578-8559-b00a5393fb67/volumes"
Jul 19 16:48:42.707464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:42.707435    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-nvpc8"
Jul 19 16:48:43.714754 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:48:43.714727    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7mcg4"
Jul 19 16:49:09.748570 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:09.748538    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-txgvt"
Jul 19 16:49:11.756513 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:11.756480    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2hmq5"
Jul 19 16:49:12.749398 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:12.749363    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:12.799923 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:12.799894    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:25.245001 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.244960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:25.357648 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.357618    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:25.381845 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.381813    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rxxl"]
Jul 19 16:49:25.382106 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.382095    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449756a7-c36e-4578-8559-b00a5393fb67" containerName="discovery"
Jul 19 16:49:25.382156 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.382107    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="449756a7-c36e-4578-8559-b00a5393fb67" containerName="discovery"
Jul 19 16:49:25.382198 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.382157    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="449756a7-c36e-4578-8559-b00a5393fb67" containerName="discovery"
Jul 19 16:49:25.387236 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.387205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:25.389726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.389703    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-7bfrn\""
Jul 19 16:49:25.389726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.389721    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 19 16:49:25.389925 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.389745    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 19 16:49:25.395729 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.395708    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rxxl"]
Jul 19 16:49:25.421529 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.421492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:25.475745 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.475717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:25.530529 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.530451    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zn8\" (UniqueName: \"kubernetes.io/projected/03b7f426-5998-4b35-afed-4d60999edf32-kube-api-access-56zn8\") pod \"kuadrant-operator-catalog-4rxxl\" (UID: \"03b7f426-5998-4b35-afed-4d60999edf32\") " pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:25.631201 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.631162    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zn8\" (UniqueName: \"kubernetes.io/projected/03b7f426-5998-4b35-afed-4d60999edf32-kube-api-access-56zn8\") pod \"kuadrant-operator-catalog-4rxxl\" (UID: \"03b7f426-5998-4b35-afed-4d60999edf32\") " pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:25.639189 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.639157    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zn8\" (UniqueName: \"kubernetes.io/projected/03b7f426-5998-4b35-afed-4d60999edf32-kube-api-access-56zn8\") pod \"kuadrant-operator-catalog-4rxxl\" (UID: \"03b7f426-5998-4b35-afed-4d60999edf32\") " pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:25.698363 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.698323    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:25.733759 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.733729    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rxxl"]
Jul 19 16:49:25.830087 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.830063    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rxxl"]
Jul 19 16:49:25.832610 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:49:25.832582    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b7f426_5998_4b35_afed_4d60999edf32.slice/crio-14e4c6ab003b1a44f553222c57ea863bfb9d799254869fccafc0c61843814d0c WatchSource:0}: Error finding container 14e4c6ab003b1a44f553222c57ea863bfb9d799254869fccafc0c61843814d0c: Status 404 returned error can't find the container with id 14e4c6ab003b1a44f553222c57ea863bfb9d799254869fccafc0c61843814d0c
Jul 19 16:49:25.935169 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.935135    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl" event={"ID":"03b7f426-5998-4b35-afed-4d60999edf32","Type":"ContainerStarted","Data":"14e4c6ab003b1a44f553222c57ea863bfb9d799254869fccafc0c61843814d0c"}
Jul 19 16:49:25.943376 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.943347    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7rtxz"]
Jul 19 16:49:25.946011 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.945995    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:25.953564 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:25.953540    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7rtxz"]
Jul 19 16:49:26.034343 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.034221    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqkbg\" (UniqueName: \"kubernetes.io/projected/a2841520-adfd-456a-8e78-48251f084716-kube-api-access-kqkbg\") pod \"kuadrant-operator-catalog-7rtxz\" (UID: \"a2841520-adfd-456a-8e78-48251f084716\") " pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:26.135466 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.135400    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqkbg\" (UniqueName: \"kubernetes.io/projected/a2841520-adfd-456a-8e78-48251f084716-kube-api-access-kqkbg\") pod \"kuadrant-operator-catalog-7rtxz\" (UID: \"a2841520-adfd-456a-8e78-48251f084716\") " pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:26.144167 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.144139    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqkbg\" (UniqueName: \"kubernetes.io/projected/a2841520-adfd-456a-8e78-48251f084716-kube-api-access-kqkbg\") pod \"kuadrant-operator-catalog-7rtxz\" (UID: \"a2841520-adfd-456a-8e78-48251f084716\") " pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:26.167555 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.167520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:26.255775 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.255737    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:26.370106 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.370077    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:26.407038 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.406840    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7rtxz"]
Jul 19 16:49:26.447414 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:49:26.447368    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2841520_adfd_456a_8e78_48251f084716.slice/crio-40de81322ec75a383bf78472f9bb9f4602b0ba4d4b41e3801e3cd0293fd106c4 WatchSource:0}: Error finding container 40de81322ec75a383bf78472f9bb9f4602b0ba4d4b41e3801e3cd0293fd106c4: Status 404 returned error can't find the container with id 40de81322ec75a383bf78472f9bb9f4602b0ba4d4b41e3801e3cd0293fd106c4
Jul 19 16:49:26.945807 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:26.945766    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz" event={"ID":"a2841520-adfd-456a-8e78-48251f084716","Type":"ContainerStarted","Data":"40de81322ec75a383bf78472f9bb9f4602b0ba4d4b41e3801e3cd0293fd106c4"}
Jul 19 16:49:27.339900 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.339835    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:27.387839 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.387800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:27.952887 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.952845    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl" event={"ID":"03b7f426-5998-4b35-afed-4d60999edf32","Type":"ContainerStarted","Data":"6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214"}
Jul 19 16:49:27.953083 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.952898    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl" podUID="03b7f426-5998-4b35-afed-4d60999edf32" containerName="registry-server" containerID="cri-o://6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214" gracePeriod=2
Jul 19 16:49:27.954642 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.954612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz" event={"ID":"a2841520-adfd-456a-8e78-48251f084716","Type":"ContainerStarted","Data":"7b8ece5c331f9e751f900195b24df071078df26c2ef784e9a8c6f8c586f08311"}
Jul 19 16:49:27.967451 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.967374    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl" podStartSLOduration=1.000955446 podStartE2EDuration="2.967355662s" totalImagesPullingTime="1.966400216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:49:27.966743259 +0000 UTC m=+677.871994696" watchObservedRunningTime="2026-07-19 16:49:27.967355662 +0000 UTC m=+677.872607087"
Jul 19 16:49:27.980726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:27.980673    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz" podStartSLOduration=1.628525787 podStartE2EDuration="2.980655121s" totalImagesPullingTime="1.352129334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:49:27.979831388 +0000 UTC m=+677.885082815" watchObservedRunningTime="2026-07-19 16:49:27.980655121 +0000 UTC m=+677.885906536"
Jul 19 16:49:28.205443 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.205419    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:28.355795 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.355708    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56zn8\" (UniqueName: \"kubernetes.io/projected/03b7f426-5998-4b35-afed-4d60999edf32-kube-api-access-56zn8\") pod \"03b7f426-5998-4b35-afed-4d60999edf32\" (UID: \"03b7f426-5998-4b35-afed-4d60999edf32\") "
Jul 19 16:49:28.358027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.358001    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b7f426-5998-4b35-afed-4d60999edf32-kube-api-access-56zn8" (OuterVolumeSpecName: "kube-api-access-56zn8") pod "03b7f426-5998-4b35-afed-4d60999edf32" (UID: "03b7f426-5998-4b35-afed-4d60999edf32"). InnerVolumeSpecName "kube-api-access-56zn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:49:28.456447 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.456408    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zn8\" (UniqueName: \"kubernetes.io/projected/03b7f426-5998-4b35-afed-4d60999edf32-kube-api-access-56zn8\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:49:28.960495 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.960457    2584 generic.go:356] "Generic (PLEG): container finished" podID="03b7f426-5998-4b35-afed-4d60999edf32" containerID="6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214" exitCode=0
Jul 19 16:49:28.960699 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.960521    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl"
Jul 19 16:49:28.960699 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.960547    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl" event={"ID":"03b7f426-5998-4b35-afed-4d60999edf32","Type":"ContainerDied","Data":"6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214"}
Jul 19 16:49:28.960699 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.960638    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rxxl" event={"ID":"03b7f426-5998-4b35-afed-4d60999edf32","Type":"ContainerDied","Data":"14e4c6ab003b1a44f553222c57ea863bfb9d799254869fccafc0c61843814d0c"}
Jul 19 16:49:28.960699 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.960667    2584 scope.go:117] "RemoveContainer" containerID="6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214"
Jul 19 16:49:28.969477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.969458    2584 scope.go:117] "RemoveContainer" containerID="6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214"
Jul 19 16:49:28.969809 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:49:28.969777    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214\": container with ID starting with 6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214 not found: ID does not exist" containerID="6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214"
Jul 19 16:49:28.969917 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.969815    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214"} err="failed to get container status \"6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214\": rpc error: code = NotFound desc = could not find container \"6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214\": container with ID starting with 6da550947174665791ac166614d624e22c8ba6c2d4c42ad43d1fd52359ba8214 not found: ID does not exist"
Jul 19 16:49:28.975501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.975466    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rxxl"]
Jul 19 16:49:28.979539 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:28.979517    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rxxl"]
Jul 19 16:49:30.632679 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:30.632650    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b7f426-5998-4b35-afed-4d60999edf32" path="/var/lib/kubelet/pods/03b7f426-5998-4b35-afed-4d60999edf32/volumes"
Jul 19 16:49:36.256066 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:36.256026    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:36.256526 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:36.256092    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:36.278703 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:36.278676    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:37.016435 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:37.016409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7rtxz"
Jul 19 16:49:40.140161 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:40.140129    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:46.945757 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:46.945727    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:46.998033 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:46.998002    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:47.088499 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:47.088468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:47.136991 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:47.136959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:47.687948 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:47.687913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:47.789071 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:47.789034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:49.018442 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:49.018406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:49.818244 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:49.818212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:53.069028 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.068997    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"]
Jul 19 16:49:53.069428 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.069292    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03b7f426-5998-4b35-afed-4d60999edf32" containerName="registry-server"
Jul 19 16:49:53.069428 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.069303    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="03b7f426-5998-4b35-afed-4d60999edf32" containerName="registry-server"
Jul 19 16:49:53.069428 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.069353    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="03b7f426-5998-4b35-afed-4d60999edf32" containerName="registry-server"
Jul 19 16:49:53.072037 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.072021    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:49:53.074269 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.074246    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-njxw8\""
Jul 19 16:49:53.080758 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.080735    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"]
Jul 19 16:49:53.141664 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.141622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghpz\" (UniqueName: \"kubernetes.io/projected/5f34d589-30e8-4149-a2b9-3b63964aa9dd-kube-api-access-qghpz\") pod \"authorino-operator-75cc8d5cb7-gtq29\" (UID: \"5f34d589-30e8-4149-a2b9-3b63964aa9dd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:49:53.243161 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.243131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghpz\" (UniqueName: \"kubernetes.io/projected/5f34d589-30e8-4149-a2b9-3b63964aa9dd-kube-api-access-qghpz\") pod \"authorino-operator-75cc8d5cb7-gtq29\" (UID: \"5f34d589-30e8-4149-a2b9-3b63964aa9dd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:49:53.255561 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.255531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghpz\" (UniqueName: \"kubernetes.io/projected/5f34d589-30e8-4149-a2b9-3b63964aa9dd-kube-api-access-qghpz\") pod \"authorino-operator-75cc8d5cb7-gtq29\" (UID: \"5f34d589-30e8-4149-a2b9-3b63964aa9dd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:49:53.383175 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.383083    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:49:53.524735 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:53.524701    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"]
Jul 19 16:49:53.527840 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:49:53.527815    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f34d589_30e8_4149_a2b9_3b63964aa9dd.slice/crio-856608cca7dbfdd55b74dc597a10d61d428d8af74958fa242d530abf28980534 WatchSource:0}: Error finding container 856608cca7dbfdd55b74dc597a10d61d428d8af74958fa242d530abf28980534: Status 404 returned error can't find the container with id 856608cca7dbfdd55b74dc597a10d61d428d8af74958fa242d530abf28980534
Jul 19 16:49:54.024583 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:54.024552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:54.061745 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:54.061715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29" event={"ID":"5f34d589-30e8-4149-a2b9-3b63964aa9dd","Type":"ContainerStarted","Data":"856608cca7dbfdd55b74dc597a10d61d428d8af74958fa242d530abf28980534"}
Jul 19 16:49:55.666762 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:55.666528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:56.072623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:56.072526    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29" event={"ID":"5f34d589-30e8-4149-a2b9-3b63964aa9dd","Type":"ContainerStarted","Data":"6c4b74ecbe687e8dbd1ccdf8d24da37089e8bd8c61bcf88d6beb70af5d1f5ea0"}
Jul 19 16:49:56.072623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:56.072599    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:49:56.096477 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:56.096415    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29" podStartSLOduration=0.889960754 podStartE2EDuration="3.096394055s" totalImagesPullingTime="2.206433301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:49:56.096010111 +0000 UTC m=+706.001261535" watchObservedRunningTime="2026-07-19 16:49:56.096394055 +0000 UTC m=+706.001645480"
Jul 19 16:49:57.851940 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:57.851904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:57.957808 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:57.957772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:57.993129 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:57.993091    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"]
Jul 19 16:49:57.999834 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:57.999818    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.002149 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.002125    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zf5vv\""
Jul 19 16:49:58.005347 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.005298    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"]
Jul 19 16:49:58.087218 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.087189    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1105c834-eb17-4d68-9979-d1528cf4ccd9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.087218 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.087234    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg26h\" (UniqueName: \"kubernetes.io/projected/1105c834-eb17-4d68-9979-d1528cf4ccd9-kube-api-access-vg26h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.188609 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.188579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1105c834-eb17-4d68-9979-d1528cf4ccd9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.188796 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.188621    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg26h\" (UniqueName: \"kubernetes.io/projected/1105c834-eb17-4d68-9979-d1528cf4ccd9-kube-api-access-vg26h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.189053 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.189030    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1105c834-eb17-4d68-9979-d1528cf4ccd9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.197977 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.197951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg26h\" (UniqueName: \"kubernetes.io/projected/1105c834-eb17-4d68-9979-d1528cf4ccd9-kube-api-access-vg26h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.298950 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.298921    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:58.310907 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.310873    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:49:58.449167 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:58.449142    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"]
Jul 19 16:49:58.451893 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:49:58.451866    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1105c834_eb17_4d68_9979_d1528cf4ccd9.slice/crio-1bb7224b141ff67adf088ee234dccb638c818e092be65f26da6ae5eaa52ba1c7 WatchSource:0}: Error finding container 1bb7224b141ff67adf088ee234dccb638c818e092be65f26da6ae5eaa52ba1c7: Status 404 returned error can't find the container with id 1bb7224b141ff67adf088ee234dccb638c818e092be65f26da6ae5eaa52ba1c7
Jul 19 16:49:59.069979 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:59.069933    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:49:59.086533 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:49:59.086502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98" event={"ID":"1105c834-eb17-4d68-9979-d1528cf4ccd9","Type":"ContainerStarted","Data":"1bb7224b141ff67adf088ee234dccb638c818e092be65f26da6ae5eaa52ba1c7"}
Jul 19 16:50:00.809023 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:00.808985    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"]
Jul 19 16:50:00.812224 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:00.812206    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:00.814870 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:00.814835    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 19 16:50:00.814992 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:00.814880    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mk5f7\""
Jul 19 16:50:00.822585 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:00.822555    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"]
Jul 19 16:50:00.914921 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:00.914880    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4f4\" (UniqueName: \"kubernetes.io/projected/c49a1761-a9a9-46d9-8d25-41b7318d2b49-kube-api-access-vb4f4\") pod \"dns-operator-controller-manager-864dd78c4c-fb4zt\" (UID: \"c49a1761-a9a9-46d9-8d25-41b7318d2b49\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:01.015823 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:01.015782    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4f4\" (UniqueName: \"kubernetes.io/projected/c49a1761-a9a9-46d9-8d25-41b7318d2b49-kube-api-access-vb4f4\") pod \"dns-operator-controller-manager-864dd78c4c-fb4zt\" (UID: \"c49a1761-a9a9-46d9-8d25-41b7318d2b49\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:01.026770 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:01.026738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4f4\" (UniqueName: \"kubernetes.io/projected/c49a1761-a9a9-46d9-8d25-41b7318d2b49-kube-api-access-vb4f4\") pod \"dns-operator-controller-manager-864dd78c4c-fb4zt\" (UID: \"c49a1761-a9a9-46d9-8d25-41b7318d2b49\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:01.123438 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:01.123346    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:01.278134 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:01.278108    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"]
Jul 19 16:50:01.280832 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:01.280801    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49a1761_a9a9_46d9_8d25_41b7318d2b49.slice/crio-e09e5a70df4cc212a48095b816492174beea93f525b9326bf8069259bcb6f472 WatchSource:0}: Error finding container e09e5a70df4cc212a48095b816492174beea93f525b9326bf8069259bcb6f472: Status 404 returned error can't find the container with id e09e5a70df4cc212a48095b816492174beea93f525b9326bf8069259bcb6f472
Jul 19 16:50:01.443297 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:01.443246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:01.667461 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:01.667428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:02.101538 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:02.101506    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt" event={"ID":"c49a1761-a9a9-46d9-8d25-41b7318d2b49","Type":"ContainerStarted","Data":"e09e5a70df4cc212a48095b816492174beea93f525b9326bf8069259bcb6f472"}
Jul 19 16:50:03.413296 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.413248    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"]
Jul 19 16:50:03.416783 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.416764    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:03.419346 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.419328    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8gd6q\""
Jul 19 16:50:03.426253 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.426229    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"]
Jul 19 16:50:03.539173 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.539134    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w26s\" (UniqueName: \"kubernetes.io/projected/48d353c5-ea37-4010-b74e-6e14e27921b9-kube-api-access-7w26s\") pod \"limitador-operator-controller-manager-859d7b78bd-jgv97\" (UID: \"48d353c5-ea37-4010-b74e-6e14e27921b9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:03.640612 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.640569    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w26s\" (UniqueName: \"kubernetes.io/projected/48d353c5-ea37-4010-b74e-6e14e27921b9-kube-api-access-7w26s\") pod \"limitador-operator-controller-manager-859d7b78bd-jgv97\" (UID: \"48d353c5-ea37-4010-b74e-6e14e27921b9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:03.650260 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.650218    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w26s\" (UniqueName: \"kubernetes.io/projected/48d353c5-ea37-4010-b74e-6e14e27921b9-kube-api-access-7w26s\") pod \"limitador-operator-controller-manager-859d7b78bd-jgv97\" (UID: \"48d353c5-ea37-4010-b74e-6e14e27921b9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:03.729577 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:03.729497    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:04.154027 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:04.153983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:04.544083 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:04.544061    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"]
Jul 19 16:50:04.546385 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:04.546352    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d353c5_ea37_4010_b74e_6e14e27921b9.slice/crio-2921410000dea79cb76abc8de132ec1d37b70f8b451afdc7a27d2caddc016b93 WatchSource:0}: Error finding container 2921410000dea79cb76abc8de132ec1d37b70f8b451afdc7a27d2caddc016b93: Status 404 returned error can't find the container with id 2921410000dea79cb76abc8de132ec1d37b70f8b451afdc7a27d2caddc016b93
Jul 19 16:50:05.118501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.118461    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt" event={"ID":"c49a1761-a9a9-46d9-8d25-41b7318d2b49","Type":"ContainerStarted","Data":"c4b2a674bdd4aca3a52dae3da763a1b7e139e58cf96b5efab0c44f392fff89c6"}
Jul 19 16:50:05.118725 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.118657    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:05.119836 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.119802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97" event={"ID":"48d353c5-ea37-4010-b74e-6e14e27921b9","Type":"ContainerStarted","Data":"2921410000dea79cb76abc8de132ec1d37b70f8b451afdc7a27d2caddc016b93"}
Jul 19 16:50:05.121456 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.121430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98" event={"ID":"1105c834-eb17-4d68-9979-d1528cf4ccd9","Type":"ContainerStarted","Data":"411d4522e2b27f343a17955232a1dc8e5a9943115b60169eba48b06424a2e1c9"}
Jul 19 16:50:05.121582 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.121573    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:05.136171 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.136106    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt" podStartSLOduration=2.008944391 podStartE2EDuration="5.136092003s" totalImagesPullingTime="3.127147612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:05.13466455 +0000 UTC m=+715.039915976" watchObservedRunningTime="2026-07-19 16:50:05.136092003 +0000 UTC m=+715.041343427"
Jul 19 16:50:05.151596 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.151545    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98" podStartSLOduration=2.194164148 podStartE2EDuration="8.151527739s" totalImagesPullingTime="5.957363591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:05.150671839 +0000 UTC m=+715.055923263" watchObservedRunningTime="2026-07-19 16:50:05.151527739 +0000 UTC m=+715.056779162"
Jul 19 16:50:05.234682 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.234643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:05.332349 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.332256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:05.628166 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.628079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:05.811676 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:05.811637    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:07.079747 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:07.079715    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-gtq29"
Jul 19 16:50:07.133531 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:07.133484    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97" event={"ID":"48d353c5-ea37-4010-b74e-6e14e27921b9","Type":"ContainerStarted","Data":"74b27639d3dff7bba3d15a540d608ad0adca9c4217d48dca66568490da889b04"}
Jul 19 16:50:07.133712 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:07.133582    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:07.149062 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:07.149010    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97" podStartSLOduration=2.582459183 podStartE2EDuration="4.148993618s" totalImagesPullingTime="1.566534435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:07.148537897 +0000 UTC m=+717.053789338" watchObservedRunningTime="2026-07-19 16:50:07.148993618 +0000 UTC m=+717.054245042"
Jul 19 16:50:07.234101 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:07.234065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:07.489573 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:07.489537    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:09.226960 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:09.226931    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:09.327408 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:09.327369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:16.129146 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:16.129111    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-fb4zt"
Jul 19 16:50:16.129552 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:16.129311    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:16.239994 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:16.239927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:16.341075 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:16.341032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:17.936067 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.936026    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"]
Jul 19 16:50:17.936573 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.936252    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" containerName="manager" containerID="cri-o://411d4522e2b27f343a17955232a1dc8e5a9943115b60169eba48b06424a2e1c9" gracePeriod=2
Jul 19 16:50:17.945362 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.945113    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"]
Jul 19 16:50:17.947947 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:17.947907    2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1105c834-eb17-4d68-9979-d1528cf4ccd9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-n7t98\": pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:17.955466 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.955434    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"]
Jul 19 16:50:17.955722 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.955671    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" containerName="manager" containerID="cri-o://74b27639d3dff7bba3d15a540d608ad0adca9c4217d48dca66568490da889b04" gracePeriod=2
Jul 19 16:50:17.958242 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.958118    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:17.959076 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:17.959038    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:17.959198 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.959072    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"]
Jul 19 16:50:17.959576 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.959557    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" containerName="manager"
Jul 19 16:50:17.959576 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.959578    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" containerName="manager"
Jul 19 16:50:17.959706 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.959677    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" containerName="manager"
Jul 19 16:50:17.962786 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.962767    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:17.966835 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.966810    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"]
Jul 19 16:50:17.972522 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.972500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"]
Jul 19 16:50:17.978241 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.978220    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"]
Jul 19 16:50:17.978552 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.978539    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" containerName="manager"
Jul 19 16:50:17.978596 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.978554    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" containerName="manager"
Jul 19 16:50:17.978636 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.978605    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" containerName="manager"
Jul 19 16:50:17.981412 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.981396    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:17.992923 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:17.992876    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:17.994930 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:17.994898    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgv97\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:17.995768 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:17.995744    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"]
Jul 19 16:50:17.996828 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:17.996799    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:17.998693 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:17.998661    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgv97\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:18.065534 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.065503    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c85c796-9ec9-4d64-adab-cd136f05a3b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dgrjc\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.065676 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.065554    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpcg\" (UniqueName: \"kubernetes.io/projected/8840e359-eab4-4c53-9290-d88dab31c522-kube-api-access-5kpcg\") pod \"limitador-operator-controller-manager-859d7b78bd-l8bdr\" (UID: \"8840e359-eab4-4c53-9290-d88dab31c522\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:18.065676 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.065601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfff9\" (UniqueName: \"kubernetes.io/projected/6c85c796-9ec9-4d64-adab-cd136f05a3b7-kube-api-access-gfff9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dgrjc\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.087856 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.087579    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"]
Jul 19 16:50:18.093366 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.093341    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.096976 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:18.096920    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:18.098867 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:18.098832    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgv97\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:18.104221 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.104183    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"]
Jul 19 16:50:18.166889 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.166850    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c85c796-9ec9-4d64-adab-cd136f05a3b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dgrjc\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.167070 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.166904    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgvj\" (UniqueName: \"kubernetes.io/projected/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-kube-api-access-blgvj\") pod \"kuadrant-operator-controller-manager-86878c769f-8gp4b\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.167070 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.167007    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpcg\" (UniqueName: \"kubernetes.io/projected/8840e359-eab4-4c53-9290-d88dab31c522-kube-api-access-5kpcg\") pod \"limitador-operator-controller-manager-859d7b78bd-l8bdr\" (UID: \"8840e359-eab4-4c53-9290-d88dab31c522\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:18.167173 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.167069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8gp4b\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.167173 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.167147    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfff9\" (UniqueName: \"kubernetes.io/projected/6c85c796-9ec9-4d64-adab-cd136f05a3b7-kube-api-access-gfff9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dgrjc\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.167338 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.167318    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c85c796-9ec9-4d64-adab-cd136f05a3b7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dgrjc\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.175259 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.175224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpcg\" (UniqueName: \"kubernetes.io/projected/8840e359-eab4-4c53-9290-d88dab31c522-kube-api-access-5kpcg\") pod \"limitador-operator-controller-manager-859d7b78bd-l8bdr\" (UID: \"8840e359-eab4-4c53-9290-d88dab31c522\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:18.175384 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.175263    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfff9\" (UniqueName: \"kubernetes.io/projected/6c85c796-9ec9-4d64-adab-cd136f05a3b7-kube-api-access-gfff9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dgrjc\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.181513 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.181486    2584 generic.go:356] "Generic (PLEG): container finished" podID="1105c834-eb17-4d68-9979-d1528cf4ccd9" containerID="411d4522e2b27f343a17955232a1dc8e5a9943115b60169eba48b06424a2e1c9" exitCode=0
Jul 19 16:50:18.183315 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.183272    2584 generic.go:356] "Generic (PLEG): container finished" podID="48d353c5-ea37-4010-b74e-6e14e27921b9" containerID="74b27639d3dff7bba3d15a540d608ad0adca9c4217d48dca66568490da889b04" exitCode=0
Jul 19 16:50:18.209922 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.209890    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:18.212144 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:18.212112    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:18.213891 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:18.213867    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgv97\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:18.214540 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.214526    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:18.216338 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:18.216315    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-n7t98\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:18.217958 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:18.217940    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jgv97\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:18.268411 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.268374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgvj\" (UniqueName: \"kubernetes.io/projected/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-kube-api-access-blgvj\") pod \"kuadrant-operator-controller-manager-86878c769f-8gp4b\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.268555 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.268427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8gp4b\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.268769 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.268754    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8gp4b\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.279707 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.279674    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgvj\" (UniqueName: \"kubernetes.io/projected/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-kube-api-access-blgvj\") pod \"kuadrant-operator-controller-manager-86878c769f-8gp4b\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.369421 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.369387    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w26s\" (UniqueName: \"kubernetes.io/projected/48d353c5-ea37-4010-b74e-6e14e27921b9-kube-api-access-7w26s\") pod \"48d353c5-ea37-4010-b74e-6e14e27921b9\" (UID: \"48d353c5-ea37-4010-b74e-6e14e27921b9\") "
Jul 19 16:50:18.369563 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.369466    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1105c834-eb17-4d68-9979-d1528cf4ccd9-extensions-socket-volume\") pod \"1105c834-eb17-4d68-9979-d1528cf4ccd9\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") "
Jul 19 16:50:18.369563 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.369510    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg26h\" (UniqueName: \"kubernetes.io/projected/1105c834-eb17-4d68-9979-d1528cf4ccd9-kube-api-access-vg26h\") pod \"1105c834-eb17-4d68-9979-d1528cf4ccd9\" (UID: \"1105c834-eb17-4d68-9979-d1528cf4ccd9\") "
Jul 19 16:50:18.369832 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.369802    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1105c834-eb17-4d68-9979-d1528cf4ccd9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1105c834-eb17-4d68-9979-d1528cf4ccd9" (UID: "1105c834-eb17-4d68-9979-d1528cf4ccd9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 16:50:18.371684 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.371659    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d353c5-ea37-4010-b74e-6e14e27921b9-kube-api-access-7w26s" (OuterVolumeSpecName: "kube-api-access-7w26s") pod "48d353c5-ea37-4010-b74e-6e14e27921b9" (UID: "48d353c5-ea37-4010-b74e-6e14e27921b9"). InnerVolumeSpecName "kube-api-access-7w26s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:50:18.371784 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.371699    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1105c834-eb17-4d68-9979-d1528cf4ccd9-kube-api-access-vg26h" (OuterVolumeSpecName: "kube-api-access-vg26h") pod "1105c834-eb17-4d68-9979-d1528cf4ccd9" (UID: "1105c834-eb17-4d68-9979-d1528cf4ccd9"). InnerVolumeSpecName "kube-api-access-vg26h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:50:18.383911 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.383884    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:18.393091 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.393067    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:18.407464 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.407436    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:18.470787 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.470708    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg26h\" (UniqueName: \"kubernetes.io/projected/1105c834-eb17-4d68-9979-d1528cf4ccd9-kube-api-access-vg26h\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:50:18.470787 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.470744    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w26s\" (UniqueName: \"kubernetes.io/projected/48d353c5-ea37-4010-b74e-6e14e27921b9-kube-api-access-7w26s\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:50:18.470787 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.470759    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1105c834-eb17-4d68-9979-d1528cf4ccd9-extensions-socket-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:50:18.548214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.548184    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"]
Jul 19 16:50:18.550965 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:18.550922    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c85c796_9ec9_4d64_adab_cd136f05a3b7.slice/crio-4fa66b67ca9e382edcdc6e9ef0b68da3b3b20ed2d0776a39cae0006d6407abc1 WatchSource:0}: Error finding container 4fa66b67ca9e382edcdc6e9ef0b68da3b3b20ed2d0776a39cae0006d6407abc1: Status 404 returned error can't find the container with id 4fa66b67ca9e382edcdc6e9ef0b68da3b3b20ed2d0776a39cae0006d6407abc1
Jul 19 16:50:18.567421 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.567386    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"]
Jul 19 16:50:18.570163 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:18.570140    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8840e359_eab4_4c53_9290_d88dab31c522.slice/crio-6bbb3db087dad601ae6d14403bd6b812e7ca05ab4d4c52f01c31e8248867f653 WatchSource:0}: Error finding container 6bbb3db087dad601ae6d14403bd6b812e7ca05ab4d4c52f01c31e8248867f653: Status 404 returned error can't find the container with id 6bbb3db087dad601ae6d14403bd6b812e7ca05ab4d4c52f01c31e8248867f653
Jul 19 16:50:18.590966 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.590937    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"]
Jul 19 16:50:18.601578 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:18.601554    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea092d81_63d5_4a0b_a147_c6a99cb4bc88.slice/crio-87d520d778292acd900d377efd837b648c6085b7f03dd92f310d7e66a0856517 WatchSource:0}: Error finding container 87d520d778292acd900d377efd837b648c6085b7f03dd92f310d7e66a0856517: Status 404 returned error can't find the container with id 87d520d778292acd900d377efd837b648c6085b7f03dd92f310d7e66a0856517
Jul 19 16:50:18.633191 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.633159    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1105c834-eb17-4d68-9979-d1528cf4ccd9" path="/var/lib/kubelet/pods/1105c834-eb17-4d68-9979-d1528cf4ccd9/volumes"
Jul 19 16:50:18.633811 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:18.633574    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48d353c5-ea37-4010-b74e-6e14e27921b9" path="/var/lib/kubelet/pods/48d353c5-ea37-4010-b74e-6e14e27921b9/volumes"
Jul 19 16:50:19.189600 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.189566    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jgv97"
Jul 19 16:50:19.190060 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.189620    2584 scope.go:117] "RemoveContainer" containerID="74b27639d3dff7bba3d15a540d608ad0adca9c4217d48dca66568490da889b04"
Jul 19 16:50:19.191723 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.191692    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr" event={"ID":"8840e359-eab4-4c53-9290-d88dab31c522","Type":"ContainerStarted","Data":"2b50d35c93d9ce82371686d6e5bf280871f26c27b4b4dbad1ebd7cb4944386d3"}
Jul 19 16:50:19.191843 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.191755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr" event={"ID":"8840e359-eab4-4c53-9290-d88dab31c522","Type":"ContainerStarted","Data":"6bbb3db087dad601ae6d14403bd6b812e7ca05ab4d4c52f01c31e8248867f653"}
Jul 19 16:50:19.191904 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.191871    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:19.193663 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.193640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b" event={"ID":"ea092d81-63d5-4a0b-a147-c6a99cb4bc88","Type":"ContainerStarted","Data":"fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619"}
Jul 19 16:50:19.193778 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.193677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b" event={"ID":"ea092d81-63d5-4a0b-a147-c6a99cb4bc88","Type":"ContainerStarted","Data":"87d520d778292acd900d377efd837b648c6085b7f03dd92f310d7e66a0856517"}
Jul 19 16:50:19.193778 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.193697    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:19.195562 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.195531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc" event={"ID":"6c85c796-9ec9-4d64-adab-cd136f05a3b7","Type":"ContainerStarted","Data":"8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548"}
Jul 19 16:50:19.195662 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.195585    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc" event={"ID":"6c85c796-9ec9-4d64-adab-cd136f05a3b7","Type":"ContainerStarted","Data":"4fa66b67ca9e382edcdc6e9ef0b68da3b3b20ed2d0776a39cae0006d6407abc1"}
Jul 19 16:50:19.195662 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.195627    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:19.196889 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.196873    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n7t98"
Jul 19 16:50:19.199576 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.199555    2584 scope.go:117] "RemoveContainer" containerID="411d4522e2b27f343a17955232a1dc8e5a9943115b60169eba48b06424a2e1c9"
Jul 19 16:50:19.208675 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.208617    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr" podStartSLOduration=2.208601016 podStartE2EDuration="2.208601016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:19.20621766 +0000 UTC m=+729.111469082" watchObservedRunningTime="2026-07-19 16:50:19.208601016 +0000 UTC m=+729.113852434"
Jul 19 16:50:19.223552 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.223474    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b" podStartSLOduration=1.223459595 podStartE2EDuration="1.223459595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:19.221905193 +0000 UTC m=+729.127156618" watchObservedRunningTime="2026-07-19 16:50:19.223459595 +0000 UTC m=+729.128711018"
Jul 19 16:50:19.239339 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:19.239292    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc" podStartSLOduration=2.239262959 podStartE2EDuration="2.239262959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:19.237658007 +0000 UTC m=+729.142909433" watchObservedRunningTime="2026-07-19 16:50:19.239262959 +0000 UTC m=+729.144514457"
Jul 19 16:50:20.879685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:20.879652    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:20.990690 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:20.990656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:24.663332 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:24.663298    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:24.761928 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:24.761898    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:24.862131 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:24.862099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:24.964814 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:24.964732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:27.375016 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:27.374975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:27.474849 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:27.474818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:30.205179 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.205152    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-l8bdr"
Jul 19 16:50:30.205581 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.205231    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:30.205581 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.205268    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 16:50:30.298200 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.298165    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"]
Jul 19 16:50:30.298436 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.298409    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc" podUID="6c85c796-9ec9-4d64-adab-cd136f05a3b7" containerName="manager" containerID="cri-o://8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548" gracePeriod=10
Jul 19 16:50:30.553060 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.552053    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:30.576561 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.576519    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfff9\" (UniqueName: \"kubernetes.io/projected/6c85c796-9ec9-4d64-adab-cd136f05a3b7-kube-api-access-gfff9\") pod \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") "
Jul 19 16:50:30.576750 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.576580    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c85c796-9ec9-4d64-adab-cd136f05a3b7-extensions-socket-volume\") pod \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\" (UID: \"6c85c796-9ec9-4d64-adab-cd136f05a3b7\") "
Jul 19 16:50:30.577111 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.577076    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c85c796-9ec9-4d64-adab-cd136f05a3b7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6c85c796-9ec9-4d64-adab-cd136f05a3b7" (UID: "6c85c796-9ec9-4d64-adab-cd136f05a3b7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 16:50:30.579163 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.579124    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c85c796-9ec9-4d64-adab-cd136f05a3b7-kube-api-access-gfff9" (OuterVolumeSpecName: "kube-api-access-gfff9") pod "6c85c796-9ec9-4d64-adab-cd136f05a3b7" (UID: "6c85c796-9ec9-4d64-adab-cd136f05a3b7"). InnerVolumeSpecName "kube-api-access-gfff9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:50:30.677792 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.677749    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfff9\" (UniqueName: \"kubernetes.io/projected/6c85c796-9ec9-4d64-adab-cd136f05a3b7-kube-api-access-gfff9\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:50:30.677792 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:30.677784    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c85c796-9ec9-4d64-adab-cd136f05a3b7-extensions-socket-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:50:31.253884 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.253852    2584 generic.go:356] "Generic (PLEG): container finished" podID="6c85c796-9ec9-4d64-adab-cd136f05a3b7" containerID="8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548" exitCode=0
Jul 19 16:50:31.254399 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.253897    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"
Jul 19 16:50:31.254399 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.253911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc" event={"ID":"6c85c796-9ec9-4d64-adab-cd136f05a3b7","Type":"ContainerDied","Data":"8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548"}
Jul 19 16:50:31.254399 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.253954    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc" event={"ID":"6c85c796-9ec9-4d64-adab-cd136f05a3b7","Type":"ContainerDied","Data":"4fa66b67ca9e382edcdc6e9ef0b68da3b3b20ed2d0776a39cae0006d6407abc1"}
Jul 19 16:50:31.254399 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.253976    2584 scope.go:117] "RemoveContainer" containerID="8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548"
Jul 19 16:50:31.262440 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.262422    2584 scope.go:117] "RemoveContainer" containerID="8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548"
Jul 19 16:50:31.262750 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:31.262723    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548\": container with ID starting with 8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548 not found: ID does not exist" containerID="8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548"
Jul 19 16:50:31.262842 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.262757    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548"} err="failed to get container status \"8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548\": rpc error: code = NotFound desc = could not find container \"8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548\": container with ID starting with 8c3a372c351876b87092f101a6f358a34e8555cbdf9ac0959855281c12e85548 not found: ID does not exist"
Jul 19 16:50:31.277523 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.277493    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"]
Jul 19 16:50:31.283532 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:31.283507    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dgrjc"]
Jul 19 16:50:32.631931 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:32.631897    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c85c796-9ec9-4d64-adab-cd136f05a3b7" path="/var/lib/kubelet/pods/6c85c796-9ec9-4d64-adab-cd136f05a3b7/volumes"
Jul 19 16:50:33.860936 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:33.860901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:33.913001 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:33.912967    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:38.613266 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:38.613231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:38.713607 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:38.713575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:42.420389 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:42.420348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:42.521778 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:42.521748    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:43.012724 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:43.012684    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:43.069965 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:43.069934    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:46.644082 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:46.644051    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:46.744404 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:46.744375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:47.043922 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:47.043876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:48.492180 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.492149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:48.502798 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.502773    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:50:48.503121 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.503108    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c85c796-9ec9-4d64-adab-cd136f05a3b7" containerName="manager"
Jul 19 16:50:48.503178 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.503123    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c85c796-9ec9-4d64-adab-cd136f05a3b7" containerName="manager"
Jul 19 16:50:48.503214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.503201    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c85c796-9ec9-4d64-adab-cd136f05a3b7" containerName="manager"
Jul 19 16:50:48.505803 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.505788    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.508245 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.508209    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 19 16:50:48.508394 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.508372    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vkgtk\""
Jul 19 16:50:48.516887 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.516860    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:50:48.597710 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.597671    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:50:48.606909 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.606872    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:48.627014 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.626982    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmbc\" (UniqueName: \"kubernetes.io/projected/3b67e66d-c116-4f06-acdd-cb5ac78103eb-kube-api-access-gxmbc\") pod \"limitador-limitador-64c94fd64f-m9qb7\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.627183 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.627030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b67e66d-c116-4f06-acdd-cb5ac78103eb-config-file\") pod \"limitador-limitador-64c94fd64f-m9qb7\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.657694 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.657661    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:48.728506 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.728463    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmbc\" (UniqueName: \"kubernetes.io/projected/3b67e66d-c116-4f06-acdd-cb5ac78103eb-kube-api-access-gxmbc\") pod \"limitador-limitador-64c94fd64f-m9qb7\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.728506 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.728510    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b67e66d-c116-4f06-acdd-cb5ac78103eb-config-file\") pod \"limitador-limitador-64c94fd64f-m9qb7\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.729208 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.729173    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b67e66d-c116-4f06-acdd-cb5ac78103eb-config-file\") pod \"limitador-limitador-64c94fd64f-m9qb7\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.740177 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.740146    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmbc\" (UniqueName: \"kubernetes.io/projected/3b67e66d-c116-4f06-acdd-cb5ac78103eb-kube-api-access-gxmbc\") pod \"limitador-limitador-64c94fd64f-m9qb7\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.815455 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.815319    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:48.951006 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:48.950941    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:50:48.953253 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:48.953218    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b67e66d_c116_4f06_acdd_cb5ac78103eb.slice/crio-8137cb392eec6704fb4588a20b4edea22ae5cee45baea171bd8be84c88aee367 WatchSource:0}: Error finding container 8137cb392eec6704fb4588a20b4edea22ae5cee45baea171bd8be84c88aee367: Status 404 returned error can't find the container with id 8137cb392eec6704fb4588a20b4edea22ae5cee45baea171bd8be84c88aee367
Jul 19 16:50:49.277580 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.277547    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-7rjm7"]
Jul 19 16:50:49.280425 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.280390    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:49.282853 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.282832    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b9dtc\""
Jul 19 16:50:49.287562 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.287537    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7rjm7"]
Jul 19 16:50:49.318695 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.318664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:49.333820 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.333783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7" event={"ID":"3b67e66d-c116-4f06-acdd-cb5ac78103eb","Type":"ContainerStarted","Data":"8137cb392eec6704fb4588a20b4edea22ae5cee45baea171bd8be84c88aee367"}
Jul 19 16:50:49.334062 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.334044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nt48\" (UniqueName: \"kubernetes.io/projected/3b4facaa-85ba-4378-868e-b8a701400101-kube-api-access-5nt48\") pod \"authorino-58cd955675-7rjm7\" (UID: \"3b4facaa-85ba-4378-868e-b8a701400101\") " pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:49.419942 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.419913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:49.435117 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.435087    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nt48\" (UniqueName: \"kubernetes.io/projected/3b4facaa-85ba-4378-868e-b8a701400101-kube-api-access-5nt48\") pod \"authorino-58cd955675-7rjm7\" (UID: \"3b4facaa-85ba-4378-868e-b8a701400101\") " pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:49.448096 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.448063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nt48\" (UniqueName: \"kubernetes.io/projected/3b4facaa-85ba-4378-868e-b8a701400101-kube-api-access-5nt48\") pod \"authorino-58cd955675-7rjm7\" (UID: \"3b4facaa-85ba-4378-868e-b8a701400101\") " pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:49.591642 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.591554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:49.783866 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:49.783803    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-7rjm7"]
Jul 19 16:50:49.787444 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:50:49.787409    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b4facaa_85ba_4378_868e_b8a701400101.slice/crio-98d7bad4f8d4d88fe12845cb433326a4f8668b18de22a7f597b87f883cf297c7 WatchSource:0}: Error finding container 98d7bad4f8d4d88fe12845cb433326a4f8668b18de22a7f597b87f883cf297c7: Status 404 returned error can't find the container with id 98d7bad4f8d4d88fe12845cb433326a4f8668b18de22a7f597b87f883cf297c7
Jul 19 16:50:50.340588 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:50.340551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7rjm7" event={"ID":"3b4facaa-85ba-4378-868e-b8a701400101","Type":"ContainerStarted","Data":"98d7bad4f8d4d88fe12845cb433326a4f8668b18de22a7f597b87f883cf297c7"}
Jul 19 16:50:51.102695 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:51.102655    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:51.152697 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:51.152668    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:52.355965 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:52.355928    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7" event={"ID":"3b67e66d-c116-4f06-acdd-cb5ac78103eb","Type":"ContainerStarted","Data":"b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5"}
Jul 19 16:50:52.356452 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:52.356095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:50:52.373546 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:52.373497    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7" podStartSLOduration=1.566236717 podStartE2EDuration="4.373475679s" totalImagesPullingTime="2.807238962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:52.371444839 +0000 UTC m=+762.276696265" watchObservedRunningTime="2026-07-19 16:50:52.373475679 +0000 UTC m=+762.278727104"
Jul 19 16:50:52.869685 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:52.869642    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:52.967800 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:52.967772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:54.367186 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:54.367152    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7rjm7" event={"ID":"3b4facaa-85ba-4378-868e-b8a701400101","Type":"ContainerStarted","Data":"73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44"}
Jul 19 16:50:54.381967 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:54.381915    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-7rjm7" podStartSLOduration=1.528299658 podStartE2EDuration="5.381896933s" totalImagesPullingTime="3.853597275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:50:54.380311368 +0000 UTC m=+764.285562788" watchObservedRunningTime="2026-07-19 16:50:54.381896933 +0000 UTC m=+764.287148357"
Jul 19 16:50:56.064467 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:56.064426    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7rjm7"]
Jul 19 16:50:56.376701 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:56.376616    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-7rjm7" podUID="3b4facaa-85ba-4378-868e-b8a701400101" containerName="authorino" containerID="cri-o://73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44" gracePeriod=30
Jul 19 16:50:56.629516 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:56.629457    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:56.701609 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:56.701574    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nt48\" (UniqueName: \"kubernetes.io/projected/3b4facaa-85ba-4378-868e-b8a701400101-kube-api-access-5nt48\") pod \"3b4facaa-85ba-4378-868e-b8a701400101\" (UID: \"3b4facaa-85ba-4378-868e-b8a701400101\") "
Jul 19 16:50:56.703746 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:56.703720    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4facaa-85ba-4378-868e-b8a701400101-kube-api-access-5nt48" (OuterVolumeSpecName: "kube-api-access-5nt48") pod "3b4facaa-85ba-4378-868e-b8a701400101" (UID: "3b4facaa-85ba-4378-868e-b8a701400101"). InnerVolumeSpecName "kube-api-access-5nt48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:50:56.802972 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:56.802922    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nt48\" (UniqueName: \"kubernetes.io/projected/3b4facaa-85ba-4378-868e-b8a701400101-kube-api-access-5nt48\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:50:57.384025 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.382782    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b4facaa-85ba-4378-868e-b8a701400101" containerID="73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44" exitCode=0
Jul 19 16:50:57.384025 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.382945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7rjm7" event={"ID":"3b4facaa-85ba-4378-868e-b8a701400101","Type":"ContainerDied","Data":"73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44"}
Jul 19 16:50:57.384025 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.382994    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-7rjm7" event={"ID":"3b4facaa-85ba-4378-868e-b8a701400101","Type":"ContainerDied","Data":"98d7bad4f8d4d88fe12845cb433326a4f8668b18de22a7f597b87f883cf297c7"}
Jul 19 16:50:57.384025 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.383018    2584 scope.go:117] "RemoveContainer" containerID="73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44"
Jul 19 16:50:57.384025 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.383217    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-7rjm7"
Jul 19 16:50:57.395589 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.395570    2584 scope.go:117] "RemoveContainer" containerID="73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44"
Jul 19 16:50:57.395867 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:50:57.395845    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44\": container with ID starting with 73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44 not found: ID does not exist" containerID="73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44"
Jul 19 16:50:57.395953 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.395880    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44"} err="failed to get container status \"73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44\": rpc error: code = NotFound desc = could not find container \"73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44\": container with ID starting with 73f79a021025a29ae6d319c105df2badfa13ea7482b9e59ef058c9a31c750b44 not found: ID does not exist"
Jul 19 16:50:57.404923 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.404896    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-7rjm7"]
Jul 19 16:50:57.408562 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.408539    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-7rjm7"]
Jul 19 16:50:57.637708 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.637624    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:57.738793 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:57.738761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:50:58.631797 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:50:58.631760    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b4facaa-85ba-4378-868e-b8a701400101" path="/var/lib/kubelet/pods/3b4facaa-85ba-4378-868e-b8a701400101/volumes"
Jul 19 16:51:03.362709 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:03.362675    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:51:07.072298 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.072240    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:51:07.072736 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.072480    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7" podUID="3b67e66d-c116-4f06-acdd-cb5ac78103eb" containerName="limitador" containerID="cri-o://b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5" gracePeriod=30
Jul 19 16:51:07.625849 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.625824    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:51:07.693864 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.693831    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b67e66d-c116-4f06-acdd-cb5ac78103eb-config-file\") pod \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") "
Jul 19 16:51:07.694036 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.693885    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxmbc\" (UniqueName: \"kubernetes.io/projected/3b67e66d-c116-4f06-acdd-cb5ac78103eb-kube-api-access-gxmbc\") pod \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\" (UID: \"3b67e66d-c116-4f06-acdd-cb5ac78103eb\") "
Jul 19 16:51:07.694199 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.694175    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b67e66d-c116-4f06-acdd-cb5ac78103eb-config-file" (OuterVolumeSpecName: "config-file") pod "3b67e66d-c116-4f06-acdd-cb5ac78103eb" (UID: "3b67e66d-c116-4f06-acdd-cb5ac78103eb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 16:51:07.696112 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.696084    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b67e66d-c116-4f06-acdd-cb5ac78103eb-kube-api-access-gxmbc" (OuterVolumeSpecName: "kube-api-access-gxmbc") pod "3b67e66d-c116-4f06-acdd-cb5ac78103eb" (UID: "3b67e66d-c116-4f06-acdd-cb5ac78103eb"). InnerVolumeSpecName "kube-api-access-gxmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:51:07.794958 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.794927    2584 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3b67e66d-c116-4f06-acdd-cb5ac78103eb-config-file\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:51:07.794958 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:07.794954    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmbc\" (UniqueName: \"kubernetes.io/projected/3b67e66d-c116-4f06-acdd-cb5ac78103eb-kube-api-access-gxmbc\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:51:08.439244 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.439198    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b67e66d-c116-4f06-acdd-cb5ac78103eb" containerID="b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5" exitCode=0
Jul 19 16:51:08.439761 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.439252    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"
Jul 19 16:51:08.439761 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.439258    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7" event={"ID":"3b67e66d-c116-4f06-acdd-cb5ac78103eb","Type":"ContainerDied","Data":"b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5"}
Jul 19 16:51:08.439761 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.439333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-m9qb7" event={"ID":"3b67e66d-c116-4f06-acdd-cb5ac78103eb","Type":"ContainerDied","Data":"8137cb392eec6704fb4588a20b4edea22ae5cee45baea171bd8be84c88aee367"}
Jul 19 16:51:08.439761 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.439356    2584 scope.go:117] "RemoveContainer" containerID="b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5"
Jul 19 16:51:08.448252 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.448235    2584 scope.go:117] "RemoveContainer" containerID="b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5"
Jul 19 16:51:08.448555 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:51:08.448537    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5\": container with ID starting with b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5 not found: ID does not exist" containerID="b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5"
Jul 19 16:51:08.448610 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.448565    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5"} err="failed to get container status \"b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5\": rpc error: code = NotFound desc = could not find container \"b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5\": container with ID starting with b63dfc270939ae05f40a753a66ca4dee6e52550d47b488d9eafc4e79af9f56f5 not found: ID does not exist"
Jul 19 16:51:08.458028 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.457918    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:51:08.459428 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.459409    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-m9qb7"]
Jul 19 16:51:08.632233 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:08.632194    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b67e66d-c116-4f06-acdd-cb5ac78103eb" path="/var/lib/kubelet/pods/3b67e66d-c116-4f06-acdd-cb5ac78103eb/volumes"
Jul 19 16:51:09.714157 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:09.714124    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:09.812409 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:09.812379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:10.161209 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161174    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"]
Jul 19 16:51:10.161791 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161768    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b67e66d-c116-4f06-acdd-cb5ac78103eb" containerName="limitador"
Jul 19 16:51:10.161791 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161791    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b67e66d-c116-4f06-acdd-cb5ac78103eb" containerName="limitador"
Jul 19 16:51:10.161985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161831    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4facaa-85ba-4378-868e-b8a701400101" containerName="authorino"
Jul 19 16:51:10.161985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161840    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4facaa-85ba-4378-868e-b8a701400101" containerName="authorino"
Jul 19 16:51:10.161985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161908    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b67e66d-c116-4f06-acdd-cb5ac78103eb" containerName="limitador"
Jul 19 16:51:10.161985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.161920    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4facaa-85ba-4378-868e-b8a701400101" containerName="authorino"
Jul 19 16:51:10.166659 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.166641    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.169236 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.169204    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 19 16:51:10.169426 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.169204    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 16:51:10.169985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.169965    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-z5d5f\""
Jul 19 16:51:10.170081 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.169989    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 19 16:51:10.170818 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.170793    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"]
Jul 19 16:51:10.188957 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.188928    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:10.318898 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.318856    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwznc\" (UniqueName: \"kubernetes.io/projected/5f5c12d7-f2c5-45fe-85c8-eca5979713c9-kube-api-access-qwznc\") pod \"postgres-69fb4594cf-7vjxm\" (UID: \"5f5c12d7-f2c5-45fe-85c8-eca5979713c9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.319080 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.318918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f5c12d7-f2c5-45fe-85c8-eca5979713c9-data\") pod \"postgres-69fb4594cf-7vjxm\" (UID: \"5f5c12d7-f2c5-45fe-85c8-eca5979713c9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.420270 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.420168    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwznc\" (UniqueName: \"kubernetes.io/projected/5f5c12d7-f2c5-45fe-85c8-eca5979713c9-kube-api-access-qwznc\") pod \"postgres-69fb4594cf-7vjxm\" (UID: \"5f5c12d7-f2c5-45fe-85c8-eca5979713c9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.420270 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.420220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f5c12d7-f2c5-45fe-85c8-eca5979713c9-data\") pod \"postgres-69fb4594cf-7vjxm\" (UID: \"5f5c12d7-f2c5-45fe-85c8-eca5979713c9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.420596 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.420577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5f5c12d7-f2c5-45fe-85c8-eca5979713c9-data\") pod \"postgres-69fb4594cf-7vjxm\" (UID: \"5f5c12d7-f2c5-45fe-85c8-eca5979713c9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.428428 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.428407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwznc\" (UniqueName: \"kubernetes.io/projected/5f5c12d7-f2c5-45fe-85c8-eca5979713c9-kube-api-access-qwznc\") pod \"postgres-69fb4594cf-7vjxm\" (UID: \"5f5c12d7-f2c5-45fe-85c8-eca5979713c9\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.478790 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.478756    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:10.619546 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:10.619519    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"]
Jul 19 16:51:10.621881 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:51:10.621855    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5c12d7_f2c5_45fe_85c8_eca5979713c9.slice/crio-3d621532c4a611e086b3cfabf721084c4c140c219eaacc7ea777f524fc5ab1d8 WatchSource:0}: Error finding container 3d621532c4a611e086b3cfabf721084c4c140c219eaacc7ea777f524fc5ab1d8: Status 404 returned error can't find the container with id 3d621532c4a611e086b3cfabf721084c4c140c219eaacc7ea777f524fc5ab1d8
Jul 19 16:51:11.458968 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:11.458929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm" event={"ID":"5f5c12d7-f2c5-45fe-85c8-eca5979713c9","Type":"ContainerStarted","Data":"3d621532c4a611e086b3cfabf721084c4c140c219eaacc7ea777f524fc5ab1d8"}
Jul 19 16:51:15.409994 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:15.409972    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 19 16:51:16.488560 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:16.488522    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm" event={"ID":"5f5c12d7-f2c5-45fe-85c8-eca5979713c9","Type":"ContainerStarted","Data":"f78c273dd4eec5f947a0c2e97bae68be7da0d3d1280ab52497b7b728ace94db1"}
Jul 19 16:51:16.489026 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:16.488691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:16.503939 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:16.503879    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm" podStartSLOduration=1.719830996 podStartE2EDuration="6.503864393s" totalImagesPullingTime="4.784033397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:51:16.502464993 +0000 UTC m=+786.407716452" watchObservedRunningTime="2026-07-19 16:51:16.503864393 +0000 UTC m=+786.409115817"
Jul 19 16:51:22.522269 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:22.522241    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7vjxm"
Jul 19 16:51:22.833911 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:22.833806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:23.002423 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.002391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:23.060813 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.060781    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ssj6q"]
Jul 19 16:51:23.068847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.068822    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:23.069094 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.069072    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ssj6q"]
Jul 19 16:51:23.070991 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.070972    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b9dtc\""
Jul 19 16:51:23.234953 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.234921    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55np5\" (UniqueName: \"kubernetes.io/projected/6d1df5ea-6974-4972-b786-32150cb95fa9-kube-api-access-55np5\") pod \"authorino-5d4b67c446-ssj6q\" (UID: \"6d1df5ea-6974-4972-b786-32150cb95fa9\") " pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:23.290861 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.290828    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ssj6q"]
Jul 19 16:51:23.291100 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:51:23.291079    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-55np5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-ssj6q" podUID="6d1df5ea-6974-4972-b786-32150cb95fa9"
Jul 19 16:51:23.311742 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.311708    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c666b6b64-xkzj9"]
Jul 19 16:51:23.326550 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.326516    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c666b6b64-xkzj9"]
Jul 19 16:51:23.326698 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.326635    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:23.335504 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.335479    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55np5\" (UniqueName: \"kubernetes.io/projected/6d1df5ea-6974-4972-b786-32150cb95fa9-kube-api-access-55np5\") pod \"authorino-5d4b67c446-ssj6q\" (UID: \"6d1df5ea-6974-4972-b786-32150cb95fa9\") " pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:23.345445 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.345422    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55np5\" (UniqueName: \"kubernetes.io/projected/6d1df5ea-6974-4972-b786-32150cb95fa9-kube-api-access-55np5\") pod \"authorino-5d4b67c446-ssj6q\" (UID: \"6d1df5ea-6974-4972-b786-32150cb95fa9\") " pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:23.436120 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.436081    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8wp\" (UniqueName: \"kubernetes.io/projected/637f12ee-6ae8-458c-acba-78ad19e91ee0-kube-api-access-dm8wp\") pod \"authorino-6c666b6b64-xkzj9\" (UID: \"637f12ee-6ae8-458c-acba-78ad19e91ee0\") " pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:23.480085 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.480047    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c666b6b64-xkzj9"]
Jul 19 16:51:23.480359 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:51:23.480317    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dm8wp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6c666b6b64-xkzj9" podUID="637f12ee-6ae8-458c-acba-78ad19e91ee0"
Jul 19 16:51:23.507389 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.507308    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56b5495544-vlq6z"]
Jul 19 16:51:23.519095 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.519067    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:23.519263 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.519067    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:23.534829 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.534805    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b5495544-vlq6z"]
Jul 19 16:51:23.535186 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.534908    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.536773 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.536747    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8wp\" (UniqueName: \"kubernetes.io/projected/637f12ee-6ae8-458c-acba-78ad19e91ee0-kube-api-access-dm8wp\") pod \"authorino-6c666b6b64-xkzj9\" (UID: \"637f12ee-6ae8-458c-acba-78ad19e91ee0\") " pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:23.536922 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.536873    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 19 16:51:23.538582 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.538560    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:23.543859 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.543836    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8wp\" (UniqueName: \"kubernetes.io/projected/637f12ee-6ae8-458c-acba-78ad19e91ee0-kube-api-access-dm8wp\") pod \"authorino-6c666b6b64-xkzj9\" (UID: \"637f12ee-6ae8-458c-acba-78ad19e91ee0\") " pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:23.558894 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.558860    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:23.637765 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.637732    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55np5\" (UniqueName: \"kubernetes.io/projected/6d1df5ea-6974-4972-b786-32150cb95fa9-kube-api-access-55np5\") pod \"6d1df5ea-6974-4972-b786-32150cb95fa9\" (UID: \"6d1df5ea-6974-4972-b786-32150cb95fa9\") "
Jul 19 16:51:23.637956 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.637863    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2tzg\" (UniqueName: \"kubernetes.io/projected/10e2c47b-e4d4-41f8-9548-b4493702e75c-kube-api-access-j2tzg\") pod \"authorino-56b5495544-vlq6z\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") " pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.637956 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.637916    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10e2c47b-e4d4-41f8-9548-b4493702e75c-tls-cert\") pod \"authorino-56b5495544-vlq6z\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") " pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.640109 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.640082    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1df5ea-6974-4972-b786-32150cb95fa9-kube-api-access-55np5" (OuterVolumeSpecName: "kube-api-access-55np5") pod "6d1df5ea-6974-4972-b786-32150cb95fa9" (UID: "6d1df5ea-6974-4972-b786-32150cb95fa9"). InnerVolumeSpecName "kube-api-access-55np5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:51:23.738529 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.738487    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm8wp\" (UniqueName: \"kubernetes.io/projected/637f12ee-6ae8-458c-acba-78ad19e91ee0-kube-api-access-dm8wp\") pod \"637f12ee-6ae8-458c-acba-78ad19e91ee0\" (UID: \"637f12ee-6ae8-458c-acba-78ad19e91ee0\") "
Jul 19 16:51:23.738720 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.738619    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10e2c47b-e4d4-41f8-9548-b4493702e75c-tls-cert\") pod \"authorino-56b5495544-vlq6z\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") " pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.738789 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.738726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2tzg\" (UniqueName: \"kubernetes.io/projected/10e2c47b-e4d4-41f8-9548-b4493702e75c-kube-api-access-j2tzg\") pod \"authorino-56b5495544-vlq6z\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") " pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.738789 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.738784    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55np5\" (UniqueName: \"kubernetes.io/projected/6d1df5ea-6974-4972-b786-32150cb95fa9-kube-api-access-55np5\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:51:23.740738 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.740709    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637f12ee-6ae8-458c-acba-78ad19e91ee0-kube-api-access-dm8wp" (OuterVolumeSpecName: "kube-api-access-dm8wp") pod "637f12ee-6ae8-458c-acba-78ad19e91ee0" (UID: "637f12ee-6ae8-458c-acba-78ad19e91ee0"). InnerVolumeSpecName "kube-api-access-dm8wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:51:23.741058 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.741039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10e2c47b-e4d4-41f8-9548-b4493702e75c-tls-cert\") pod \"authorino-56b5495544-vlq6z\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") " pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.746643 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.746617    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2tzg\" (UniqueName: \"kubernetes.io/projected/10e2c47b-e4d4-41f8-9548-b4493702e75c-kube-api-access-j2tzg\") pod \"authorino-56b5495544-vlq6z\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") " pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.839467 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.839384    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8wp\" (UniqueName: \"kubernetes.io/projected/637f12ee-6ae8-458c-acba-78ad19e91ee0-kube-api-access-dm8wp\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:51:23.844269 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.844248    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:51:23.999430 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:23.999408    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56b5495544-vlq6z"]
Jul 19 16:51:24.001551 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:51:24.001523    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10e2c47b_e4d4_41f8_9548_b4493702e75c.slice/crio-3fa92c56f608fdba5a954272295a8d64d82e1ec86d2fa6cae3282a9f1b01717a WatchSource:0}: Error finding container 3fa92c56f608fdba5a954272295a8d64d82e1ec86d2fa6cae3282a9f1b01717a: Status 404 returned error can't find the container with id 3fa92c56f608fdba5a954272295a8d64d82e1ec86d2fa6cae3282a9f1b01717a
Jul 19 16:51:24.524985 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.524944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b5495544-vlq6z" event={"ID":"10e2c47b-e4d4-41f8-9548-b4493702e75c","Type":"ContainerStarted","Data":"3fa92c56f608fdba5a954272295a8d64d82e1ec86d2fa6cae3282a9f1b01717a"}
Jul 19 16:51:24.525184 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.525004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c666b6b64-xkzj9"
Jul 19 16:51:24.525184 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.525004    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ssj6q"
Jul 19 16:51:24.556606 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.556575    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c666b6b64-xkzj9"]
Jul 19 16:51:24.559575 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.559547    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c666b6b64-xkzj9"]
Jul 19 16:51:24.577699 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.577662    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ssj6q"]
Jul 19 16:51:24.582430 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.582404    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ssj6q"]
Jul 19 16:51:24.632022 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.631985    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637f12ee-6ae8-458c-acba-78ad19e91ee0" path="/var/lib/kubelet/pods/637f12ee-6ae8-458c-acba-78ad19e91ee0/volumes"
Jul 19 16:51:24.632218 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:24.632205    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d1df5ea-6974-4972-b786-32150cb95fa9" path="/var/lib/kubelet/pods/6d1df5ea-6974-4972-b786-32150cb95fa9/volumes"
Jul 19 16:51:25.532002 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:25.531965    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b5495544-vlq6z" event={"ID":"10e2c47b-e4d4-41f8-9548-b4493702e75c","Type":"ContainerStarted","Data":"7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b"}
Jul 19 16:51:25.547419 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:25.547358    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56b5495544-vlq6z" podStartSLOduration=1.8607715740000002 podStartE2EDuration="2.547335749s" totalImagesPullingTime="686.564175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:51:25.546134652 +0000 UTC m=+795.451386075" watchObservedRunningTime="2026-07-19 16:51:25.547335749 +0000 UTC m=+795.452587174"
Jul 19 16:51:30.187460 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.187411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:30.206823 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.206791    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bf9666bc9-j9pj6"]
Jul 19 16:51:30.218531 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.218500    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.220709 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.220678    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf9666bc9-j9pj6"]
Jul 19 16:51:30.221495 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.221272    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbxtn\""
Jul 19 16:51:30.221495 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.221421    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 19 16:51:30.221835 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.221819    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 19 16:51:30.288182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.288153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:30.301414 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.301389    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-webhook-cert\") pod \"maas-controller-7bf9666bc9-j9pj6\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") " pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.301524 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.301446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkjt\" (UniqueName: \"kubernetes.io/projected/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-kube-api-access-lhkjt\") pod \"maas-controller-7bf9666bc9-j9pj6\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") " pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.387373 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.387338    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:30.393505 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.393480    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645d86df7-tgs6z"]
Jul 19 16:51:30.395675 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.395655    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.401932 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.401912    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-webhook-cert\") pod \"maas-controller-7bf9666bc9-j9pj6\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") " pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.402030 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.401972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkjt\" (UniqueName: \"kubernetes.io/projected/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-kube-api-access-lhkjt\") pod \"maas-controller-7bf9666bc9-j9pj6\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") " pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.404633 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.404612    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-webhook-cert\") pod \"maas-controller-7bf9666bc9-j9pj6\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") " pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.406527 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.406502    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d86df7-tgs6z"]
Jul 19 16:51:30.415133 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.415113    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkjt\" (UniqueName: \"kubernetes.io/projected/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-kube-api-access-lhkjt\") pod \"maas-controller-7bf9666bc9-j9pj6\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") " pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.502575 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.502496    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8b2\" (UniqueName: \"kubernetes.io/projected/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-kube-api-access-8r8b2\") pod \"maas-controller-645d86df7-tgs6z\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") " pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.502575 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.502536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-webhook-cert\") pod \"maas-controller-645d86df7-tgs6z\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") " pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.530968 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.530940    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:30.603858 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.603826    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8b2\" (UniqueName: \"kubernetes.io/projected/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-kube-api-access-8r8b2\") pod \"maas-controller-645d86df7-tgs6z\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") " pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.604034 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.603868    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-webhook-cert\") pod \"maas-controller-645d86df7-tgs6z\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") " pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.606633 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.606604    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-webhook-cert\") pod \"maas-controller-645d86df7-tgs6z\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") " pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.613037 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.613008    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8b2\" (UniqueName: \"kubernetes.io/projected/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-kube-api-access-8r8b2\") pod \"maas-controller-645d86df7-tgs6z\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") " pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.668067 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.668000    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf9666bc9-j9pj6"]
Jul 19 16:51:30.670704 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:51:30.670677    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3320bf9b_96cb_4b1a_a091_ee9dcd5653a2.slice/crio-1971864a95dafb38753ecbe361a1c3fbaff7bb283ff0f8b0360732dd7985bd68 WatchSource:0}: Error finding container 1971864a95dafb38753ecbe361a1c3fbaff7bb283ff0f8b0360732dd7985bd68: Status 404 returned error can't find the container with id 1971864a95dafb38753ecbe361a1c3fbaff7bb283ff0f8b0360732dd7985bd68
Jul 19 16:51:30.705074 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.705034    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:30.839821 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:30.839796    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d86df7-tgs6z"]
Jul 19 16:51:30.841830 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:51:30.841805    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77aaaf2f_03c8_4b5c_a037_9312d84f9dd3.slice/crio-c01d4ee07fa44952b4be1592bfb4dba547d785fc1ff436cffbe09524c04324be WatchSource:0}: Error finding container c01d4ee07fa44952b4be1592bfb4dba547d785fc1ff436cffbe09524c04324be: Status 404 returned error can't find the container with id c01d4ee07fa44952b4be1592bfb4dba547d785fc1ff436cffbe09524c04324be
Jul 19 16:51:31.564429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:31.564228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6" event={"ID":"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2","Type":"ContainerStarted","Data":"1971864a95dafb38753ecbe361a1c3fbaff7bb283ff0f8b0360732dd7985bd68"}
Jul 19 16:51:31.568672 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:31.568579    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-tgs6z" event={"ID":"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3","Type":"ContainerStarted","Data":"c01d4ee07fa44952b4be1592bfb4dba547d785fc1ff436cffbe09524c04324be"}
Jul 19 16:51:33.270000 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.269961    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:33.367373 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.367262    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:33.579927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.579878    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6" event={"ID":"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2","Type":"ContainerStarted","Data":"8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66"}
Jul 19 16:51:33.580124 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.580053    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:33.581342 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.581315    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-tgs6z" event={"ID":"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3","Type":"ContainerStarted","Data":"06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77"}
Jul 19 16:51:33.581452 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.581423    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:33.597220 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.596752    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6" podStartSLOduration=1.15800787 podStartE2EDuration="3.596735451s" totalImagesPullingTime="2.438727581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:51:33.596106984 +0000 UTC m=+803.501358410" watchObservedRunningTime="2026-07-19 16:51:33.596735451 +0000 UTC m=+803.501986881"
Jul 19 16:51:33.613250 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:33.613194    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645d86df7-tgs6z" podStartSLOduration=1.335211916 podStartE2EDuration="3.613175368s" totalImagesPullingTime="2.277963452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:51:33.61158351 +0000 UTC m=+803.516834935" watchObservedRunningTime="2026-07-19 16:51:33.613175368 +0000 UTC m=+803.518426796"
Jul 19 16:51:37.832755 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:37.832713    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:37.884694 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:37.884662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:51:44.588683 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:44.588651    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:51:44.589064 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:44.588721    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:44.646532 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:44.646491    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf9666bc9-j9pj6"]
Jul 19 16:51:44.646748 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:44.646720    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6" podUID="3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" containerName="manager" containerID="cri-o://8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66" gracePeriod=30
Jul 19 16:51:44.894583 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:44.894558    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:45.029951 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.029923    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhkjt\" (UniqueName: \"kubernetes.io/projected/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-kube-api-access-lhkjt\") pod \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") "
Jul 19 16:51:45.030132 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.029993    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-webhook-cert\") pod \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\" (UID: \"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2\") "
Jul 19 16:51:45.032239 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.032203    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" (UID: "3320bf9b-96cb-4b1a-a091-ee9dcd5653a2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:51:45.032384 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.032269    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-kube-api-access-lhkjt" (OuterVolumeSpecName: "kube-api-access-lhkjt") pod "3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" (UID: "3320bf9b-96cb-4b1a-a091-ee9dcd5653a2"). InnerVolumeSpecName "kube-api-access-lhkjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:51:45.131304 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.131190    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhkjt\" (UniqueName: \"kubernetes.io/projected/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-kube-api-access-lhkjt\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:51:45.131304 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.131216    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2-webhook-cert\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:51:45.637393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.637362    2584 generic.go:356] "Generic (PLEG): container finished" podID="3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" containerID="8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66" exitCode=0
Jul 19 16:51:45.637852 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.637403    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6"
Jul 19 16:51:45.637852 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.637399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6" event={"ID":"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2","Type":"ContainerDied","Data":"8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66"}
Jul 19 16:51:45.637852 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.637531    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf9666bc9-j9pj6" event={"ID":"3320bf9b-96cb-4b1a-a091-ee9dcd5653a2","Type":"ContainerDied","Data":"1971864a95dafb38753ecbe361a1c3fbaff7bb283ff0f8b0360732dd7985bd68"}
Jul 19 16:51:45.637852 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.637567    2584 scope.go:117] "RemoveContainer" containerID="8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66"
Jul 19 16:51:45.646838 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.646822    2584 scope.go:117] "RemoveContainer" containerID="8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66"
Jul 19 16:51:45.647083 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:51:45.647066    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66\": container with ID starting with 8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66 not found: ID does not exist" containerID="8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66"
Jul 19 16:51:45.647129 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.647091    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66"} err="failed to get container status \"8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66\": rpc error: code = NotFound desc = could not find container \"8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66\": container with ID starting with 8da1bc4f9bce5b708438136a58f6f674a118ebdee460e645beba389c3b112d66 not found: ID does not exist"
Jul 19 16:51:45.658051 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.658026    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf9666bc9-j9pj6"]
Jul 19 16:51:45.661624 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:45.661600    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bf9666bc9-j9pj6"]
Jul 19 16:51:46.632402 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:51:46.632369    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" path="/var/lib/kubelet/pods/3320bf9b-96cb-4b1a-a091-ee9dcd5653a2/volumes"
Jul 19 16:52:05.095371 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.095327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:05.191703 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.191667    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:05.772560 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.772529    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:05.811414 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.811376    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"]
Jul 19 16:52:05.811730 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.811717    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" containerName="manager"
Jul 19 16:52:05.811785 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.811732    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" containerName="manager"
Jul 19 16:52:05.811820 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.811786    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3320bf9b-96cb-4b1a-a091-ee9dcd5653a2" containerName="manager"
Jul 19 16:52:05.814610 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.814589    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:05.817305 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.817260    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 19 16:52:05.817305 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.817272    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r9kbz\""
Jul 19 16:52:05.825910 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.825874    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"]
Jul 19 16:52:05.918915 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.918874    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-maas-api-tls\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:05.919091 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.918926    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-openshift-service-ca\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:05.919091 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:05.919023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzcp\" (UniqueName: \"kubernetes.io/projected/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-kube-api-access-zqzcp\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.020182 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.020145    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-openshift-service-ca\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.020379 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.020197    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzcp\" (UniqueName: \"kubernetes.io/projected/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-kube-api-access-zqzcp\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.020379 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.020246    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-maas-api-tls\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.020452 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:52:06.020385    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 19 16:52:06.020485 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:52:06.020474    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-maas-api-tls podName:ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251 nodeName:}" failed. No retries permitted until 2026-07-19 16:52:06.52043903 +0000 UTC m=+836.425690450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-maas-api-tls") pod "maas-api-84f575779b-9tbbr" (UID: "ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251") : secret "maas-api-serving-cert" not found
Jul 19 16:52:06.020823 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.020803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-openshift-service-ca\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.030862 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.030796    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzcp\" (UniqueName: \"kubernetes.io/projected/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-kube-api-access-zqzcp\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.149079 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.149044    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:06.249643 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.249614    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:06.321229 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.321150    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:06.321940 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.321920    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"]
Jul 19 16:52:06.325017 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.324998    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.327663 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.327640    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bcss9\""
Jul 19 16:52:06.327798 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.327649    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 19 16:52:06.334645 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.334620    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"]
Jul 19 16:52:06.358778 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.358750    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-424xr"]
Jul 19 16:52:06.361819 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.361802    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.371033 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.371012    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-424xr"]
Jul 19 16:52:06.423877 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.423812    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6e3017-6988-461f-ab28-4e568492b12a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xr6lv\" (UID: \"bf6e3017-6988-461f-ab28-4e568492b12a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.423877 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.423864    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbc2\" (UniqueName: \"kubernetes.io/projected/bf6e3017-6988-461f-ab28-4e568492b12a-kube-api-access-6bbc2\") pod \"payload-pre-processing-7677b774bb-xr6lv\" (UID: \"bf6e3017-6988-461f-ab28-4e568492b12a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.524792 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.524755    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9vh\" (UniqueName: \"kubernetes.io/projected/ecf689aa-0c99-4ad0-8b43-68f002aebebe-kube-api-access-rm9vh\") pod \"payload-processing-5bf47cd988-424xr\" (UID: \"ecf689aa-0c99-4ad0-8b43-68f002aebebe\") " pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.524792 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.524794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbc2\" (UniqueName: \"kubernetes.io/projected/bf6e3017-6988-461f-ab28-4e568492b12a-kube-api-access-6bbc2\") pod \"payload-pre-processing-7677b774bb-xr6lv\" (UID: \"bf6e3017-6988-461f-ab28-4e568492b12a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.524995 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.524927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf689aa-0c99-4ad0-8b43-68f002aebebe-plugins-config-volume\") pod \"payload-processing-5bf47cd988-424xr\" (UID: \"ecf689aa-0c99-4ad0-8b43-68f002aebebe\") " pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.525031 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.525002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-maas-api-tls\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.525079 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.525056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6e3017-6988-461f-ab28-4e568492b12a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xr6lv\" (UID: \"bf6e3017-6988-461f-ab28-4e568492b12a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.525750 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.525725    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6e3017-6988-461f-ab28-4e568492b12a-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xr6lv\" (UID: \"bf6e3017-6988-461f-ab28-4e568492b12a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.528185 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.528162    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251-maas-api-tls\") pod \"maas-api-84f575779b-9tbbr\" (UID: \"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251\") " pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.533258 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.533231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbc2\" (UniqueName: \"kubernetes.io/projected/bf6e3017-6988-461f-ab28-4e568492b12a-kube-api-access-6bbc2\") pod \"payload-pre-processing-7677b774bb-xr6lv\" (UID: \"bf6e3017-6988-461f-ab28-4e568492b12a\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.626517 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.626416    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9vh\" (UniqueName: \"kubernetes.io/projected/ecf689aa-0c99-4ad0-8b43-68f002aebebe-kube-api-access-rm9vh\") pod \"payload-processing-5bf47cd988-424xr\" (UID: \"ecf689aa-0c99-4ad0-8b43-68f002aebebe\") " pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.626517 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.626499    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf689aa-0c99-4ad0-8b43-68f002aebebe-plugins-config-volume\") pod \"payload-processing-5bf47cd988-424xr\" (UID: \"ecf689aa-0c99-4ad0-8b43-68f002aebebe\") " pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.627214 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.627183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf689aa-0c99-4ad0-8b43-68f002aebebe-plugins-config-volume\") pod \"payload-processing-5bf47cd988-424xr\" (UID: \"ecf689aa-0c99-4ad0-8b43-68f002aebebe\") " pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.639072 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.639035    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:06.639991 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.639959    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9vh\" (UniqueName: \"kubernetes.io/projected/ecf689aa-0c99-4ad0-8b43-68f002aebebe-kube-api-access-rm9vh\") pod \"payload-processing-5bf47cd988-424xr\" (UID: \"ecf689aa-0c99-4ad0-8b43-68f002aebebe\") " pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.672197 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.672166    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:06.726379 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.726348    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:06.801635 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.801603    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"]
Jul 19 16:52:06.809611 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:52:06.809325    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6e3017_6988_461f_ab28_4e568492b12a.slice/crio-f2d6f13b8e149b20cf1edaf33e2b621e3223199aca65be32bd2811a7aec98292 WatchSource:0}: Error finding container f2d6f13b8e149b20cf1edaf33e2b621e3223199aca65be32bd2811a7aec98292: Status 404 returned error can't find the container with id f2d6f13b8e149b20cf1edaf33e2b621e3223199aca65be32bd2811a7aec98292
Jul 19 16:52:06.853657 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.853629    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-424xr"]
Jul 19 16:52:06.856034 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:52:06.855996    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf689aa_0c99_4ad0_8b43_68f002aebebe.slice/crio-0cc6752cb5ffd3b569c343f792483295f05b3bd42468371cc81393244b4ec4c5 WatchSource:0}: Error finding container 0cc6752cb5ffd3b569c343f792483295f05b3bd42468371cc81393244b4ec4c5: Status 404 returned error can't find the container with id 0cc6752cb5ffd3b569c343f792483295f05b3bd42468371cc81393244b4ec4c5
Jul 19 16:52:06.907919 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:06.907896    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"]
Jul 19 16:52:06.910467 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:52:06.910439    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3780ba_bbbe_4a1e_9c81_cfe7c8c37251.slice/crio-7699bb046598a286078b5e008c68ab6684aa2b35021a77037ffeecb844406bd4 WatchSource:0}: Error finding container 7699bb046598a286078b5e008c68ab6684aa2b35021a77037ffeecb844406bd4: Status 404 returned error can't find the container with id 7699bb046598a286078b5e008c68ab6684aa2b35021a77037ffeecb844406bd4
Jul 19 16:52:07.748485 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:07.748410    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr" event={"ID":"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251","Type":"ContainerStarted","Data":"7699bb046598a286078b5e008c68ab6684aa2b35021a77037ffeecb844406bd4"}
Jul 19 16:52:07.750753 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:07.750680    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-424xr" event={"ID":"ecf689aa-0c99-4ad0-8b43-68f002aebebe","Type":"ContainerStarted","Data":"0cc6752cb5ffd3b569c343f792483295f05b3bd42468371cc81393244b4ec4c5"}
Jul 19 16:52:07.752179 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:07.752116    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv" event={"ID":"bf6e3017-6988-461f-ab28-4e568492b12a","Type":"ContainerStarted","Data":"f2d6f13b8e149b20cf1edaf33e2b621e3223199aca65be32bd2811a7aec98292"}
Jul 19 16:52:09.764246 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.764206    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr" event={"ID":"ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251","Type":"ContainerStarted","Data":"212d083b5c99c09ed6c4f0553f47cf7203648999e7523e87619df5bea1e05278"}
Jul 19 16:52:09.764743 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.764397    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:09.765870 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.765841    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-424xr" event={"ID":"ecf689aa-0c99-4ad0-8b43-68f002aebebe","Type":"ContainerStarted","Data":"0d3be700614fdf3105af79e2b6ccfd66b183aa4db6b9505560fc29134373f2ab"}
Jul 19 16:52:09.765994 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.765895    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:09.767290 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.767251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv" event={"ID":"bf6e3017-6988-461f-ab28-4e568492b12a","Type":"ContainerStarted","Data":"f1cdc86cc0e5cef9e1eeb7052c2571d9853bdb63cba53113efa56c964ec1e749"}
Jul 19 16:52:09.767440 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.767318    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:09.783192 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.783142    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr" podStartSLOduration=2.137509081 podStartE2EDuration="4.783126275s" totalImagesPullingTime="2.645617194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:52:09.781150349 +0000 UTC m=+839.686401773" watchObservedRunningTime="2026-07-19 16:52:09.783126275 +0000 UTC m=+839.688377699"
Jul 19 16:52:09.797354 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.797266    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv" podStartSLOduration=0.988941259 podStartE2EDuration="3.797244585s" totalImagesPullingTime="2.808303326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:52:09.795355224 +0000 UTC m=+839.700606653" watchObservedRunningTime="2026-07-19 16:52:09.797244585 +0000 UTC m=+839.702496011"
Jul 19 16:52:09.810676 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:09.810569    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-424xr" podStartSLOduration=1.043042499 podStartE2EDuration="3.810546071s" totalImagesPullingTime="2.767503572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:52:09.80975361 +0000 UTC m=+839.715005034" watchObservedRunningTime="2026-07-19 16:52:09.810546071 +0000 UTC m=+839.715797494"
Jul 19 16:52:15.776882 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:15.776848    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84f575779b-9tbbr"
Jul 19 16:52:16.743863 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:16.743823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:16.843512 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:16.843462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:18.501383 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:18.501346    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:20.773648 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:20.773596    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-424xr"
Jul 19 16:52:20.774125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:20.774098    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xr6lv"
Jul 19 16:52:28.069047 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:28.069008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:36.227810 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:36.227767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:36.274135 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:36.274098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:37.066623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:37.066589    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:46.835849 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.834971    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"]
Jul 19 16:52:46.840643 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.840615    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:46.844377 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.844350    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 19 16:52:46.844612 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.844591    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kp5v5\""
Jul 19 16:52:46.844713 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.844681    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 19 16:52:46.844713 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.844685    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 19 16:52:46.847393 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.847369    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"]
Jul 19 16:52:46.863301 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.863238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:52:46.972934 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.972896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b84d58-4032-4de6-888a-79e68ccce32f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:46.973125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.972952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:46.973125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.972972    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzl5\" (UniqueName: \"kubernetes.io/projected/04b84d58-4032-4de6-888a-79e68ccce32f-kube-api-access-wzzl5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:46.973125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.973039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:46.973125 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.973117    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:46.973337 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:46.973158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074138 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074103    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074138 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzl5\" (UniqueName: \"kubernetes.io/projected/04b84d58-4032-4de6-888a-79e68ccce32f-kube-api-access-wzzl5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074400 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074172    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074400 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074209    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074400 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074400 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074262    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b84d58-4032-4de6-888a-79e68ccce32f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074640 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074679 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074626    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.074721 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.074684    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.076688 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.076658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b84d58-4032-4de6-888a-79e68ccce32f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.076929 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.076910    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b84d58-4032-4de6-888a-79e68ccce32f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.082487 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.082460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzl5\" (UniqueName: \"kubernetes.io/projected/04b84d58-4032-4de6-888a-79e68ccce32f-kube-api-access-wzzl5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qhq8h\" (UID: \"04b84d58-4032-4de6-888a-79e68ccce32f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.154435 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.154409    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:47.292667 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.292638    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"]
Jul 19 16:52:47.294511 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:52:47.294483    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b84d58_4032_4de6_888a_79e68ccce32f.slice/crio-81f9ba964a893e259be836ac275976641a85c5610d50b97a1402a19808fd24a3 WatchSource:0}: Error finding container 81f9ba964a893e259be836ac275976641a85c5610d50b97a1402a19808fd24a3: Status 404 returned error can't find the container with id 81f9ba964a893e259be836ac275976641a85c5610d50b97a1402a19808fd24a3
Jul 19 16:52:47.953578 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:47.953544    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h" event={"ID":"04b84d58-4032-4de6-888a-79e68ccce32f","Type":"ContainerStarted","Data":"81f9ba964a893e259be836ac275976641a85c5610d50b97a1402a19808fd24a3"}
Jul 19 16:52:48.959813 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:48.959774    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h" event={"ID":"04b84d58-4032-4de6-888a-79e68ccce32f","Type":"ContainerStarted","Data":"84b4b66d1c4309c10d5a18dffac1ebfc25bd5350b10120cd01c2bee8d87e4dea"}
Jul 19 16:52:48.978310 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:48.978226    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h" podStartSLOduration=1.9411508720000001 podStartE2EDuration="2.9782071s" totalImagesPullingTime="1.037056228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:52:48.976131881 +0000 UTC m=+878.881383307" watchObservedRunningTime="2026-07-19 16:52:48.9782071 +0000 UTC m=+878.883458525"
Jul 19 16:52:57.154802 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:57.154760    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:57.155391 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:57.154824    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:57.167429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:57.167405    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:52:58.016409 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:52:58.016379    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qhq8h"
Jul 19 16:53:04.297463 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:04.297386    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:53:04.368552 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:04.368523    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:53:10.577721 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:10.577693    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:53:10.578205 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:10.577908    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:53:15.807352 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:15.807310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:53:15.857901 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:15.857868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:53:19.076728 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:19.076692    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:53:35.136847 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.136816    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"]
Jul 19 16:53:35.140682 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.140660    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.142957 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.142934    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 19 16:53:35.149029 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.148998    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"]
Jul 19 16:53:35.168509 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.168476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:53:35.209339 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.209259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.209535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.209349    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.209535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.209397    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.209535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.209422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c7d85d-d47d-46ab-94cc-95a96e858e43-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.209535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.209453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.209535 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.209501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkzc\" (UniqueName: \"kubernetes.io/projected/86c7d85d-d47d-46ab-94cc-95a96e858e43-kube-api-access-ptkzc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310300 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310300 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310538 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310538 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310335    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c7d85d-d47d-46ab-94cc-95a96e858e43-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310538 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310358    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310538 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310396    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkzc\" (UniqueName: \"kubernetes.io/projected/86c7d85d-d47d-46ab-94cc-95a96e858e43-kube-api-access-ptkzc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310730 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310696    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310787 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310758    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.310864 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.310840    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.312781 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.312762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86c7d85d-d47d-46ab-94cc-95a96e858e43-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.312922 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.312905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86c7d85d-d47d-46ab-94cc-95a96e858e43-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.317819 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.317797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkzc\" (UniqueName: \"kubernetes.io/projected/86c7d85d-d47d-46ab-94cc-95a96e858e43-kube-api-access-ptkzc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp\" (UID: \"86c7d85d-d47d-46ab-94cc-95a96e858e43\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.450999 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.450966    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:35.602198 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.602173    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"]
Jul 19 16:53:35.604023 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:53:35.603994    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86c7d85d_d47d_46ab_94cc_95a96e858e43.slice/crio-47c56d5f2f2a001803fccbadc9410701cd86c6d9fa71abdb785a7b76f93c8cfd WatchSource:0}: Error finding container 47c56d5f2f2a001803fccbadc9410701cd86c6d9fa71abdb785a7b76f93c8cfd: Status 404 returned error can't find the container with id 47c56d5f2f2a001803fccbadc9410701cd86c6d9fa71abdb785a7b76f93c8cfd
Jul 19 16:53:35.605824 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:35.605801    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 16:53:36.187671 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:36.187632    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp" event={"ID":"86c7d85d-d47d-46ab-94cc-95a96e858e43","Type":"ContainerStarted","Data":"e4f4df4667cd5413100cca623a977403bc055306d749ace01e0eb3430466ea36"}
Jul 19 16:53:36.187671 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:36.187673    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp" event={"ID":"86c7d85d-d47d-46ab-94cc-95a96e858e43","Type":"ContainerStarted","Data":"47c56d5f2f2a001803fccbadc9410701cd86c6d9fa71abdb785a7b76f93c8cfd"}
Jul 19 16:53:36.204927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:36.204870    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp" podStartSLOduration=1.02401286 podStartE2EDuration="1.204854771s" totalImagesPullingTime="180.841911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:53:36.203016792 +0000 UTC m=+926.108268216" watchObservedRunningTime="2026-07-19 16:53:36.204854771 +0000 UTC m=+926.110106196"
Jul 19 16:53:45.451350 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:45.451302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:45.451948 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:45.451364    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:45.464076 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:45.464045    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:53:46.247653 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:53:46.247617    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp"
Jul 19 16:54:58.307126 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.307048    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56b5495544-vlq6z"]
Jul 19 16:54:58.308000 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.307948    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-56b5495544-vlq6z" podUID="10e2c47b-e4d4-41f8-9548-b4493702e75c" containerName="authorino" containerID="cri-o://7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b" gracePeriod=30
Jul 19 16:54:58.572527 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.572501    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:54:58.576408 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.576382    2584 generic.go:356] "Generic (PLEG): container finished" podID="10e2c47b-e4d4-41f8-9548-b4493702e75c" containerID="7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b" exitCode=0
Jul 19 16:54:58.576547 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.576422    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56b5495544-vlq6z"
Jul 19 16:54:58.576547 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.576468    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b5495544-vlq6z" event={"ID":"10e2c47b-e4d4-41f8-9548-b4493702e75c","Type":"ContainerDied","Data":"7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b"}
Jul 19 16:54:58.576547 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.576508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56b5495544-vlq6z" event={"ID":"10e2c47b-e4d4-41f8-9548-b4493702e75c","Type":"ContainerDied","Data":"3fa92c56f608fdba5a954272295a8d64d82e1ec86d2fa6cae3282a9f1b01717a"}
Jul 19 16:54:58.576547 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.576526    2584 scope.go:117] "RemoveContainer" containerID="7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b"
Jul 19 16:54:58.585270 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.585249    2584 scope.go:117] "RemoveContainer" containerID="7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b"
Jul 19 16:54:58.585596 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:54:58.585565    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b\": container with ID starting with 7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b not found: ID does not exist" containerID="7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b"
Jul 19 16:54:58.585699 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.585607    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b"} err="failed to get container status \"7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b\": rpc error: code = NotFound desc = could not find container \"7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b\": container with ID starting with 7716ddfbe2a278d97fef0969a1c7b723d1eddebc89e9771ce353c58d8ab0ab3b not found: ID does not exist"
Jul 19 16:54:58.655450 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.655416    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2tzg\" (UniqueName: \"kubernetes.io/projected/10e2c47b-e4d4-41f8-9548-b4493702e75c-kube-api-access-j2tzg\") pod \"10e2c47b-e4d4-41f8-9548-b4493702e75c\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") "
Jul 19 16:54:58.655605 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.655533    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10e2c47b-e4d4-41f8-9548-b4493702e75c-tls-cert\") pod \"10e2c47b-e4d4-41f8-9548-b4493702e75c\" (UID: \"10e2c47b-e4d4-41f8-9548-b4493702e75c\") "
Jul 19 16:54:58.657641 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.657611    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e2c47b-e4d4-41f8-9548-b4493702e75c-kube-api-access-j2tzg" (OuterVolumeSpecName: "kube-api-access-j2tzg") pod "10e2c47b-e4d4-41f8-9548-b4493702e75c" (UID: "10e2c47b-e4d4-41f8-9548-b4493702e75c"). InnerVolumeSpecName "kube-api-access-j2tzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:54:58.667530 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.667508    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e2c47b-e4d4-41f8-9548-b4493702e75c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "10e2c47b-e4d4-41f8-9548-b4493702e75c" (UID: "10e2c47b-e4d4-41f8-9548-b4493702e75c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:54:58.757033 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.757000    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10e2c47b-e4d4-41f8-9548-b4493702e75c-tls-cert\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:54:58.757033 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.757030    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2tzg\" (UniqueName: \"kubernetes.io/projected/10e2c47b-e4d4-41f8-9548-b4493702e75c-kube-api-access-j2tzg\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:54:58.766987 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.766954    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:54:58.863498 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.863410    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:54:58.897353 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.897325    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56b5495544-vlq6z"]
Jul 19 16:54:58.904682 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:58.904657    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56b5495544-vlq6z"]
Jul 19 16:54:59.887765 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:59.887731    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:54:59.987817 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:54:59.987777    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:55:00.634555 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:00.634515    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10e2c47b-e4d4-41f8-9548-b4493702e75c" path="/var/lib/kubelet/pods/10e2c47b-e4d4-41f8-9548-b4493702e75c/volumes"
Jul 19 16:55:45.271667 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.271625    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d86df7-tgs6z"]
Jul 19 16:55:45.272429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.271897    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-645d86df7-tgs6z" podUID="77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" containerName="manager" containerID="cri-o://06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77" gracePeriod=30
Jul 19 16:55:45.526244 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.526175    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:55:45.574237 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.574192    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-webhook-cert\") pod \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") "
Jul 19 16:55:45.574505 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.574313    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r8b2\" (UniqueName: \"kubernetes.io/projected/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-kube-api-access-8r8b2\") pod \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\" (UID: \"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3\") "
Jul 19 16:55:45.576548 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.576520    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" (UID: "77aaaf2f-03c8-4b5c-a037-9312d84f9dd3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:55:45.576848 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.576824    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-kube-api-access-8r8b2" (OuterVolumeSpecName: "kube-api-access-8r8b2") pod "77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" (UID: "77aaaf2f-03c8-4b5c-a037-9312d84f9dd3"). InnerVolumeSpecName "kube-api-access-8r8b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:55:45.675881 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.675852    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-webhook-cert\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:55:45.675881 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.675877    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8b2\" (UniqueName: \"kubernetes.io/projected/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3-kube-api-access-8r8b2\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:55:45.798207 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.798119    2584 generic.go:356] "Generic (PLEG): container finished" podID="77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" containerID="06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77" exitCode=0
Jul 19 16:55:45.798207 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.798166    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-tgs6z"
Jul 19 16:55:45.798414 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.798210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-tgs6z" event={"ID":"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3","Type":"ContainerDied","Data":"06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77"}
Jul 19 16:55:45.798414 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.798265    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-tgs6z" event={"ID":"77aaaf2f-03c8-4b5c-a037-9312d84f9dd3","Type":"ContainerDied","Data":"c01d4ee07fa44952b4be1592bfb4dba547d785fc1ff436cffbe09524c04324be"}
Jul 19 16:55:45.798414 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.798303    2584 scope.go:117] "RemoveContainer" containerID="06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77"
Jul 19 16:55:45.807349 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.807324    2584 scope.go:117] "RemoveContainer" containerID="06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77"
Jul 19 16:55:45.807655 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:55:45.807634    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77\": container with ID starting with 06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77 not found: ID does not exist" containerID="06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77"
Jul 19 16:55:45.807708 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.807666    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77"} err="failed to get container status \"06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77\": rpc error: code = NotFound desc = could not find container \"06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77\": container with ID starting with 06aa300e74fde647980943778d8dd3a90ae803940323038f9dfa02da35500a77 not found: ID does not exist"
Jul 19 16:55:45.818379 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.818351    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d86df7-tgs6z"]
Jul 19 16:55:45.822362 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:45.822328    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645d86df7-tgs6z"]
Jul 19 16:55:46.236305 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:46.236245    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:55:46.633040 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:55:46.632955    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" path="/var/lib/kubelet/pods/77aaaf2f-03c8-4b5c-a037-9312d84f9dd3/volumes"
Jul 19 16:56:46.624440 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624405    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645d86df7-n8xf7"]
Jul 19 16:56:46.624927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624759    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" containerName="manager"
Jul 19 16:56:46.624927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624770    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" containerName="manager"
Jul 19 16:56:46.624927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624782    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e2c47b-e4d4-41f8-9548-b4493702e75c" containerName="authorino"
Jul 19 16:56:46.624927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624787    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e2c47b-e4d4-41f8-9548-b4493702e75c" containerName="authorino"
Jul 19 16:56:46.624927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624854    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="10e2c47b-e4d4-41f8-9548-b4493702e75c" containerName="authorino"
Jul 19 16:56:46.624927 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.624864    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="77aaaf2f-03c8-4b5c-a037-9312d84f9dd3" containerName="manager"
Jul 19 16:56:46.628296 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.628261    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.631830 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.631803    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbxtn\""
Jul 19 16:56:46.631961 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.631804    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 19 16:56:46.631961 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.631804    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 19 16:56:46.638313 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.638259    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d86df7-n8xf7"]
Jul 19 16:56:46.713417 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.713372    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntnv\" (UniqueName: \"kubernetes.io/projected/59021854-c037-4d71-9e1b-ae8c2651544a-kube-api-access-cntnv\") pod \"maas-controller-645d86df7-n8xf7\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") " pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.713606 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.713434    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59021854-c037-4d71-9e1b-ae8c2651544a-webhook-cert\") pod \"maas-controller-645d86df7-n8xf7\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") " pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.814365 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.814325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntnv\" (UniqueName: \"kubernetes.io/projected/59021854-c037-4d71-9e1b-ae8c2651544a-kube-api-access-cntnv\") pod \"maas-controller-645d86df7-n8xf7\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") " pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.814542 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.814378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59021854-c037-4d71-9e1b-ae8c2651544a-webhook-cert\") pod \"maas-controller-645d86df7-n8xf7\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") " pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.817064 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.817041    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59021854-c037-4d71-9e1b-ae8c2651544a-webhook-cert\") pod \"maas-controller-645d86df7-n8xf7\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") " pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.822456 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.822436    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntnv\" (UniqueName: \"kubernetes.io/projected/59021854-c037-4d71-9e1b-ae8c2651544a-kube-api-access-cntnv\") pod \"maas-controller-645d86df7-n8xf7\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") " pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:46.945562 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:46.945521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:47.084348 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:47.084323    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d86df7-n8xf7"]
Jul 19 16:56:47.086186 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:56:47.086151    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59021854_c037_4d71_9e1b_ae8c2651544a.slice/crio-51bba36f13cec938e9c3a1b2e1421cefae60424f3350c60a537a43fb24dbe856 WatchSource:0}: Error finding container 51bba36f13cec938e9c3a1b2e1421cefae60424f3350c60a537a43fb24dbe856: Status 404 returned error can't find the container with id 51bba36f13cec938e9c3a1b2e1421cefae60424f3350c60a537a43fb24dbe856
Jul 19 16:56:48.089188 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:48.089150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-n8xf7" event={"ID":"59021854-c037-4d71-9e1b-ae8c2651544a","Type":"ContainerStarted","Data":"154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915"}
Jul 19 16:56:48.089722 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:48.089215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-n8xf7" event={"ID":"59021854-c037-4d71-9e1b-ae8c2651544a","Type":"ContainerStarted","Data":"51bba36f13cec938e9c3a1b2e1421cefae60424f3350c60a537a43fb24dbe856"}
Jul 19 16:56:48.089722 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:48.089368    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:56:48.105532 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:48.105479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645d86df7-n8xf7" podStartSLOduration=1.696297091 podStartE2EDuration="2.105461981s" totalImagesPullingTime="409.16489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:56:48.103510364 +0000 UTC m=+1118.008761787" watchObservedRunningTime="2026-07-19 16:56:48.105461981 +0000 UTC m=+1118.010713404"
Jul 19 16:56:59.095765 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:56:59.095732    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:57:03.665162 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:03.665126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:03.766604 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:03.766569    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d86df7-n8xf7"]
Jul 19 16:57:03.766780 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:03.766765    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-645d86df7-n8xf7" podUID="59021854-c037-4d71-9e1b-ae8c2651544a" containerName="manager" containerID="cri-o://154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915" gracePeriod=30
Jul 19 16:57:04.022614 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.022586    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:57:04.165341 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.165305    2584 generic.go:356] "Generic (PLEG): container finished" podID="59021854-c037-4d71-9e1b-ae8c2651544a" containerID="154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915" exitCode=0
Jul 19 16:57:04.165621 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.165372    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-n8xf7"
Jul 19 16:57:04.165621 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.165412    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-n8xf7" event={"ID":"59021854-c037-4d71-9e1b-ae8c2651544a","Type":"ContainerDied","Data":"154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915"}
Jul 19 16:57:04.165621 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.165471    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-n8xf7" event={"ID":"59021854-c037-4d71-9e1b-ae8c2651544a","Type":"ContainerDied","Data":"51bba36f13cec938e9c3a1b2e1421cefae60424f3350c60a537a43fb24dbe856"}
Jul 19 16:57:04.165621 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.165501    2584 scope.go:117] "RemoveContainer" containerID="154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915"
Jul 19 16:57:04.172553 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.172527    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59021854-c037-4d71-9e1b-ae8c2651544a-webhook-cert\") pod \"59021854-c037-4d71-9e1b-ae8c2651544a\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") "
Jul 19 16:57:04.172663 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.172573    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cntnv\" (UniqueName: \"kubernetes.io/projected/59021854-c037-4d71-9e1b-ae8c2651544a-kube-api-access-cntnv\") pod \"59021854-c037-4d71-9e1b-ae8c2651544a\" (UID: \"59021854-c037-4d71-9e1b-ae8c2651544a\") "
Jul 19 16:57:04.174385 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.174364    2584 scope.go:117] "RemoveContainer" containerID="154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915"
Jul 19 16:57:04.174635 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.174610    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59021854-c037-4d71-9e1b-ae8c2651544a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "59021854-c037-4d71-9e1b-ae8c2651544a" (UID: "59021854-c037-4d71-9e1b-ae8c2651544a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 16:57:04.174754 ip-10-0-142-78 kubenswrapper[2584]: E0719 16:57:04.174696    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915\": container with ID starting with 154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915 not found: ID does not exist" containerID="154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915"
Jul 19 16:57:04.174754 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.174724    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915"} err="failed to get container status \"154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915\": rpc error: code = NotFound desc = could not find container \"154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915\": container with ID starting with 154bbb2924e12da27cf048d427d52e6d55fbf9025315802343bf4d656384a915 not found: ID does not exist"
Jul 19 16:57:04.174838 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.174759    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59021854-c037-4d71-9e1b-ae8c2651544a-kube-api-access-cntnv" (OuterVolumeSpecName: "kube-api-access-cntnv") pod "59021854-c037-4d71-9e1b-ae8c2651544a" (UID: "59021854-c037-4d71-9e1b-ae8c2651544a"). InnerVolumeSpecName "kube-api-access-cntnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 16:57:04.273623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.273528    2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59021854-c037-4d71-9e1b-ae8c2651544a-webhook-cert\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:57:04.273623 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.273563    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntnv\" (UniqueName: \"kubernetes.io/projected/59021854-c037-4d71-9e1b-ae8c2651544a-kube-api-access-cntnv\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 16:57:04.485463 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.485435    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-645d86df7-n8xf7"]
Jul 19 16:57:04.489224 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.489189    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-645d86df7-n8xf7"]
Jul 19 16:57:04.632262 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:04.632179    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59021854-c037-4d71-9e1b-ae8c2651544a" path="/var/lib/kubelet/pods/59021854-c037-4d71-9e1b-ae8c2651544a/volumes"
Jul 19 16:57:05.163537 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.163502    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:05.275986 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.275956    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-645d86df7-srxjv"]
Jul 19 16:57:05.276321 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.276308    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59021854-c037-4d71-9e1b-ae8c2651544a" containerName="manager"
Jul 19 16:57:05.276321 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.276322    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="59021854-c037-4d71-9e1b-ae8c2651544a" containerName="manager"
Jul 19 16:57:05.276404 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.276395    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="59021854-c037-4d71-9e1b-ae8c2651544a" containerName="manager"
Jul 19 16:57:05.287801 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.287766    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.289456 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.289429    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d86df7-srxjv"]
Jul 19 16:57:05.291501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.291469    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbxtn\""
Jul 19 16:57:05.291647 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.291469    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 19 16:57:05.291647 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.291475    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 19 16:57:05.384618 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.384579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbvm\" (UniqueName: \"kubernetes.io/projected/a464873e-dca1-470f-8258-c35484fea26c-kube-api-access-kxbvm\") pod \"maas-controller-645d86df7-srxjv\" (UID: \"a464873e-dca1-470f-8258-c35484fea26c\") " pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.384829 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.384643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a464873e-dca1-470f-8258-c35484fea26c-webhook-cert\") pod \"maas-controller-645d86df7-srxjv\" (UID: \"a464873e-dca1-470f-8258-c35484fea26c\") " pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.485871 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.485774    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbvm\" (UniqueName: \"kubernetes.io/projected/a464873e-dca1-470f-8258-c35484fea26c-kube-api-access-kxbvm\") pod \"maas-controller-645d86df7-srxjv\" (UID: \"a464873e-dca1-470f-8258-c35484fea26c\") " pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.485871 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.485829    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a464873e-dca1-470f-8258-c35484fea26c-webhook-cert\") pod \"maas-controller-645d86df7-srxjv\" (UID: \"a464873e-dca1-470f-8258-c35484fea26c\") " pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.488538 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.488510    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a464873e-dca1-470f-8258-c35484fea26c-webhook-cert\") pod \"maas-controller-645d86df7-srxjv\" (UID: \"a464873e-dca1-470f-8258-c35484fea26c\") " pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.493953 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.493929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbvm\" (UniqueName: \"kubernetes.io/projected/a464873e-dca1-470f-8258-c35484fea26c-kube-api-access-kxbvm\") pod \"maas-controller-645d86df7-srxjv\" (UID: \"a464873e-dca1-470f-8258-c35484fea26c\") " pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.600141 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.600101    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:05.738129 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:05.738103    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-645d86df7-srxjv"]
Jul 19 16:57:05.741108 ip-10-0-142-78 kubenswrapper[2584]: W0719 16:57:05.741079    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda464873e_dca1_470f_8258_c35484fea26c.slice/crio-afddb597fd04d7fa284a3f63d61855b36adb9850e381510a11bf6d9c4fa32f3b WatchSource:0}: Error finding container afddb597fd04d7fa284a3f63d61855b36adb9850e381510a11bf6d9c4fa32f3b: Status 404 returned error can't find the container with id afddb597fd04d7fa284a3f63d61855b36adb9850e381510a11bf6d9c4fa32f3b
Jul 19 16:57:06.176949 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:06.176916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-srxjv" event={"ID":"a464873e-dca1-470f-8258-c35484fea26c","Type":"ContainerStarted","Data":"18f6886e19ddb687cf6508678b5cb8fdeb095ef9eb57097d05bfdde4262292e4"}
Jul 19 16:57:06.177501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:06.176974    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-645d86df7-srxjv" event={"ID":"a464873e-dca1-470f-8258-c35484fea26c","Type":"ContainerStarted","Data":"afddb597fd04d7fa284a3f63d61855b36adb9850e381510a11bf6d9c4fa32f3b"}
Jul 19 16:57:06.177501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:06.177052    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:06.191597 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:06.191546    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-645d86df7-srxjv" podStartSLOduration=0.882072705 podStartE2EDuration="1.191523666s" totalImagesPullingTime="309.450961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 16:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 16:57:06.191517869 +0000 UTC m=+1136.096769294" watchObservedRunningTime="2026-07-19 16:57:06.191523666 +0000 UTC m=+1136.096775091"
Jul 19 16:57:17.186010 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:17.185982    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-645d86df7-srxjv"
Jul 19 16:57:32.338501 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:32.338414    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:32.397643 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:32.397606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:40.475429 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:40.475388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:40.574726 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:40.574689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:40.702488 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:40.702450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:46.502134 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:46.502095    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:46.554522 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:46.554488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:46.633345 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:46.633310    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:46.735396 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:46.735365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:46.909354 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:46.909315    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:52.668108 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:52.668072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:52.768056 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:52.768020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:59.711610 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:59.711568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:57:59.759012 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:57:59.758979    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:58:10.614546 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:58:10.614512    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:58:10.616412 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:58:10.616386    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 16:58:45.916308 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:58:45.916258    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:58:47.333085 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:58:47.333050    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 16:59:25.473581 ip-10-0-142-78 kubenswrapper[2584]: I0719 16:59:25.473508    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:00:00.144379 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.144346    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"]
Jul 19 17:00:00.147809 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.147792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"
Jul 19 17:00:00.150156 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.150135    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:00:00.157549 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.157531    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"
Jul 19 17:00:00.162429 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.162408    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"]
Jul 19 17:00:00.290937 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.290912    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"]
Jul 19 17:00:00.293630 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:00:00.293601    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153ba808_4d37_4b91_9196_cab379fc5a54.slice/crio-0e198e6e2d27c02336495d3fec30a1c6caedbc4f068e524d968a2543df9f41ce WatchSource:0}: Error finding container 0e198e6e2d27c02336495d3fec30a1c6caedbc4f068e524d968a2543df9f41ce: Status 404 returned error can't find the container with id 0e198e6e2d27c02336495d3fec30a1c6caedbc4f068e524d968a2543df9f41ce
Jul 19 17:00:00.295925 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:00.295905    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 17:00:01.025015 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:01.024982    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms" event={"ID":"153ba808-4d37-4b91-9196-cab379fc5a54","Type":"ContainerStarted","Data":"0e198e6e2d27c02336495d3fec30a1c6caedbc4f068e524d968a2543df9f41ce"}
Jul 19 17:00:06.051865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:06.051769    2584 generic.go:356] "Generic (PLEG): container finished" podID="153ba808-4d37-4b91-9196-cab379fc5a54" containerID="9f12444d1993e523efc14c0c52ab3193eff4f32eca6d67d56cc526196bdfd7d5" exitCode=0
Jul 19 17:00:06.051865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:06.051808    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms" event={"ID":"153ba808-4d37-4b91-9196-cab379fc5a54","Type":"ContainerDied","Data":"9f12444d1993e523efc14c0c52ab3193eff4f32eca6d67d56cc526196bdfd7d5"}
Jul 19 17:00:07.191613 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:07.191588    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"
Jul 19 17:00:08.062412 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:08.062383    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms"
Jul 19 17:00:08.062412 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:08.062392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741340-htmms" event={"ID":"153ba808-4d37-4b91-9196-cab379fc5a54","Type":"ContainerDied","Data":"0e198e6e2d27c02336495d3fec30a1c6caedbc4f068e524d968a2543df9f41ce"}
Jul 19 17:00:08.062628 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:00:08.062436    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e198e6e2d27c02336495d3fec30a1c6caedbc4f068e524d968a2543df9f41ce"
Jul 19 17:01:40.820005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:40.819969    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"]
Jul 19 17:01:40.820571 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:40.820251    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b" podUID="ea092d81-63d5-4a0b-a147-c6a99cb4bc88" containerName="manager" containerID="cri-o://fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619" gracePeriod=10
Jul 19 17:01:44.285133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.285109    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 17:01:44.377150 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.377056    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-extensions-socket-volume\") pod \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") "
Jul 19 17:01:44.377150 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.377103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blgvj\" (UniqueName: \"kubernetes.io/projected/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-kube-api-access-blgvj\") pod \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\" (UID: \"ea092d81-63d5-4a0b-a147-c6a99cb4bc88\") "
Jul 19 17:01:44.377492 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.377467    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ea092d81-63d5-4a0b-a147-c6a99cb4bc88" (UID: "ea092d81-63d5-4a0b-a147-c6a99cb4bc88"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:01:44.379335 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.379311    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-kube-api-access-blgvj" (OuterVolumeSpecName: "kube-api-access-blgvj") pod "ea092d81-63d5-4a0b-a147-c6a99cb4bc88" (UID: "ea092d81-63d5-4a0b-a147-c6a99cb4bc88"). InnerVolumeSpecName "kube-api-access-blgvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:01:44.477865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.477828    2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-extensions-socket-volume\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:01:44.477865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.477858    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgvj\" (UniqueName: \"kubernetes.io/projected/ea092d81-63d5-4a0b-a147-c6a99cb4bc88-kube-api-access-blgvj\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:01:44.516416 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.516383    2584 generic.go:356] "Generic (PLEG): container finished" podID="ea092d81-63d5-4a0b-a147-c6a99cb4bc88" containerID="fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619" exitCode=0
Jul 19 17:01:44.516582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.516426    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"
Jul 19 17:01:44.516582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.516457    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b" event={"ID":"ea092d81-63d5-4a0b-a147-c6a99cb4bc88","Type":"ContainerDied","Data":"fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619"}
Jul 19 17:01:44.516582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.516498    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b" event={"ID":"ea092d81-63d5-4a0b-a147-c6a99cb4bc88","Type":"ContainerDied","Data":"87d520d778292acd900d377efd837b648c6085b7f03dd92f310d7e66a0856517"}
Jul 19 17:01:44.516582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.516520    2584 scope.go:117] "RemoveContainer" containerID="fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619"
Jul 19 17:01:44.528925 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.528908    2584 scope.go:117] "RemoveContainer" containerID="fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619"
Jul 19 17:01:44.529205 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:01:44.529188    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619\": container with ID starting with fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619 not found: ID does not exist" containerID="fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619"
Jul 19 17:01:44.529266 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.529217    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619"} err="failed to get container status \"fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619\": rpc error: code = NotFound desc = could not find container \"fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619\": container with ID starting with fe87e4bd088b094c49149fa642e9907170f5fcaaee404fa6f6745a6d7405b619 not found: ID does not exist"
Jul 19 17:01:44.537490 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.537468    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"]
Jul 19 17:01:44.542095 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.542074    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gp4b"]
Jul 19 17:01:44.633147 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:01:44.633068    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea092d81-63d5-4a0b-a147-c6a99cb4bc88" path="/var/lib/kubelet/pods/ea092d81-63d5-4a0b-a147-c6a99cb4bc88/volumes"
Jul 19 17:02:41.990240 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990205    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"]
Jul 19 17:02:41.990743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990566    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea092d81-63d5-4a0b-a147-c6a99cb4bc88" containerName="manager"
Jul 19 17:02:41.990743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990579    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea092d81-63d5-4a0b-a147-c6a99cb4bc88" containerName="manager"
Jul 19 17:02:41.990743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990589    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153ba808-4d37-4b91-9196-cab379fc5a54" containerName="cleanup"
Jul 19 17:02:41.990743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990595    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="153ba808-4d37-4b91-9196-cab379fc5a54" containerName="cleanup"
Jul 19 17:02:41.990743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990657    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="153ba808-4d37-4b91-9196-cab379fc5a54" containerName="cleanup"
Jul 19 17:02:41.990743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.990665    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea092d81-63d5-4a0b-a147-c6a99cb4bc88" containerName="manager"
Jul 19 17:02:41.994454 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.994434    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:41.996937 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:41.996917    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zf5vv\""
Jul 19 17:02:42.005404 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.005377    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"]
Jul 19 17:02:42.190812 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.190769    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a897d7c-a6eb-4642-8cc9-a25380274d92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-799fl\" (UID: \"0a897d7c-a6eb-4642-8cc9-a25380274d92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.190995 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.190855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vpr\" (UniqueName: \"kubernetes.io/projected/0a897d7c-a6eb-4642-8cc9-a25380274d92-kube-api-access-49vpr\") pod \"kuadrant-operator-controller-manager-86878c769f-799fl\" (UID: \"0a897d7c-a6eb-4642-8cc9-a25380274d92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.291360 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.291242    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a897d7c-a6eb-4642-8cc9-a25380274d92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-799fl\" (UID: \"0a897d7c-a6eb-4642-8cc9-a25380274d92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.291497 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.291367    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vpr\" (UniqueName: \"kubernetes.io/projected/0a897d7c-a6eb-4642-8cc9-a25380274d92-kube-api-access-49vpr\") pod \"kuadrant-operator-controller-manager-86878c769f-799fl\" (UID: \"0a897d7c-a6eb-4642-8cc9-a25380274d92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.291644 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.291624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a897d7c-a6eb-4642-8cc9-a25380274d92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-799fl\" (UID: \"0a897d7c-a6eb-4642-8cc9-a25380274d92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.299453 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.299431    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vpr\" (UniqueName: \"kubernetes.io/projected/0a897d7c-a6eb-4642-8cc9-a25380274d92-kube-api-access-49vpr\") pod \"kuadrant-operator-controller-manager-86878c769f-799fl\" (UID: \"0a897d7c-a6eb-4642-8cc9-a25380274d92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.305312 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.305268    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.445461 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.445393    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"]
Jul 19 17:02:42.447766 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:02:42.447728    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a897d7c_a6eb_4642_8cc9_a25380274d92.slice/crio-e1e6ea8bcf41115de065f9852c1a6802520b4e970b2628861dfd2a8a3715a8bc WatchSource:0}: Error finding container e1e6ea8bcf41115de065f9852c1a6802520b4e970b2628861dfd2a8a3715a8bc: Status 404 returned error can't find the container with id e1e6ea8bcf41115de065f9852c1a6802520b4e970b2628861dfd2a8a3715a8bc
Jul 19 17:02:42.786761 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.786710    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl" event={"ID":"0a897d7c-a6eb-4642-8cc9-a25380274d92","Type":"ContainerStarted","Data":"6a1ced7a49873cdc8e8d279be9202605b8ad7b9955749d2772732eb3dc78b29b"}
Jul 19 17:02:42.786966 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.786784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl" event={"ID":"0a897d7c-a6eb-4642-8cc9-a25380274d92","Type":"ContainerStarted","Data":"e1e6ea8bcf41115de065f9852c1a6802520b4e970b2628861dfd2a8a3715a8bc"}
Jul 19 17:02:42.786966 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.786833    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:02:42.808674 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:42.808623    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl" podStartSLOduration=1.808605664 podStartE2EDuration="1.808605664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:02:42.805573391 +0000 UTC m=+1472.710824851" watchObservedRunningTime="2026-07-19 17:02:42.808605664 +0000 UTC m=+1472.713857087"
Jul 19 17:02:53.794133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:02:53.794097    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-799fl"
Jul 19 17:03:10.643100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:03:10.643074    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:03:10.646216 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:03:10.646194    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:05:21.790566 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:21.790531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:05:26.737848 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.737809    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"]
Jul 19 17:05:26.741316 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.741297    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.743786 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.743766    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-000f0eba-kserve-self-signed-certs\""
Jul 19 17:05:26.756938 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.756896    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"]
Jul 19 17:05:26.764094 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.764069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:05:26.831720 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.831683    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tmp-dir\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.831720 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.831727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knmnt\" (UniqueName: \"kubernetes.io/projected/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-kube-api-access-knmnt\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.831987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.831747    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-dshm\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.831987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.831821    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tls-certs\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.831987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.831896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-home\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.831987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.831948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-model-cache\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932416 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-model-cache\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932466    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tmp-dir\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932512    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knmnt\" (UniqueName: \"kubernetes.io/projected/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-kube-api-access-knmnt\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932539    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-dshm\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932563    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tls-certs\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-home\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932913 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-model-cache\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.932913 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.932847    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tmp-dir\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.933077 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.933055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-home\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.935001 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.934972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-dshm\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.935158 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.935139    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tls-certs\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:26.941020 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:26.940993    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knmnt\" (UniqueName: \"kubernetes.io/projected/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-kube-api-access-knmnt\") pod \"e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") " pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:27.050897 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:27.050815    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:27.190428 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:27.190402    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"]
Jul 19 17:05:27.192547 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:05:27.192518    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d86dd65_60d7_43e8_88fd_ef245dfa66c6.slice/crio-4c08ddf441fdce5cbadcf3d001f2e486b831af1a7451fea2bfec0011857a48e2 WatchSource:0}: Error finding container 4c08ddf441fdce5cbadcf3d001f2e486b831af1a7451fea2bfec0011857a48e2: Status 404 returned error can't find the container with id 4c08ddf441fdce5cbadcf3d001f2e486b831af1a7451fea2bfec0011857a48e2
Jul 19 17:05:27.604524 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:27.604488    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz" event={"ID":"6d86dd65-60d7-43e8-88fd-ef245dfa66c6","Type":"ContainerStarted","Data":"b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3"}
Jul 19 17:05:27.604524 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:27.604530    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz" event={"ID":"6d86dd65-60d7-43e8-88fd-ef245dfa66c6","Type":"ContainerStarted","Data":"4c08ddf441fdce5cbadcf3d001f2e486b831af1a7451fea2bfec0011857a48e2"}
Jul 19 17:05:27.619948 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:27.619901    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz" podStartSLOduration=1.619885691 podStartE2EDuration="1.619885691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:05:27.618746182 +0000 UTC m=+1637.523997633" watchObservedRunningTime="2026-07-19 17:05:27.619885691 +0000 UTC m=+1637.525137115"
Jul 19 17:05:31.633445 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.633400    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"]
Jul 19 17:05:31.637040 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.637017    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.639633 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.639610    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-000f0eba-kserve-self-signed-certs\""
Jul 19 17:05:31.645938 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.645898    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"]
Jul 19 17:05:31.658014 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.657986    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:05:31.674092 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.674064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-tmp-dir\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.674237 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.674100    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-home\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.674237 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.674125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a92ae97a-7755-4538-819a-f4c180932362-tls-certs\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.674237 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.674209    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-dshm\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.674365 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.674259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-model-cache\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.674365 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.674309    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgmf\" (UniqueName: \"kubernetes.io/projected/a92ae97a-7755-4538-819a-f4c180932362-kube-api-access-wtgmf\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775023 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.774988    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-tmp-dir\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775023 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-home\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775236 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775050    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a92ae97a-7755-4538-819a-f4c180932362-tls-certs\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775335 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-dshm\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775394 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775351    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-model-cache\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775394 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775384    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgmf\" (UniqueName: \"kubernetes.io/projected/a92ae97a-7755-4538-819a-f4c180932362-kube-api-access-wtgmf\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775521    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-home\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775549    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-tmp-dir\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.775833 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.775805    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-model-cache\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.777682 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.777658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-dshm\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.778056 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.778029    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a92ae97a-7755-4538-819a-f4c180932362-tls-certs\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.784334 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.784308    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgmf\" (UniqueName: \"kubernetes.io/projected/a92ae97a-7755-4538-819a-f4c180932362-kube-api-access-wtgmf\") pod \"e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") " pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:31.949100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:31.949057    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:32.090407 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:32.090385    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"]
Jul 19 17:05:32.092457 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:05:32.092427    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda92ae97a_7755_4538_819a_f4c180932362.slice/crio-0bc41d314b3193b8ec8ad37446caf9b3dadf44dceb986785a3c8d957156f9c4f WatchSource:0}: Error finding container 0bc41d314b3193b8ec8ad37446caf9b3dadf44dceb986785a3c8d957156f9c4f: Status 404 returned error can't find the container with id 0bc41d314b3193b8ec8ad37446caf9b3dadf44dceb986785a3c8d957156f9c4f
Jul 19 17:05:32.633442 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:32.633413    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4" event={"ID":"a92ae97a-7755-4538-819a-f4c180932362","Type":"ContainerStarted","Data":"e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da"}
Jul 19 17:05:32.633593 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:32.633451    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4" event={"ID":"a92ae97a-7755-4538-819a-f4c180932362","Type":"ContainerStarted","Data":"0bc41d314b3193b8ec8ad37446caf9b3dadf44dceb986785a3c8d957156f9c4f"}
Jul 19 17:05:32.648210 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:32.648161    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4" podStartSLOduration=1.648145992 podStartE2EDuration="1.648145992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:05:32.646720064 +0000 UTC m=+1642.551971489" watchObservedRunningTime="2026-07-19 17:05:32.648145992 +0000 UTC m=+1642.553397415"
Jul 19 17:05:37.051468 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:37.051430    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:37.051886 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:37.051752    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:37.064104 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:37.064077    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:37.667921 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:37.667891    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:05:38.597660 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:38.597621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:05:38.651729 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:38.651689    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:05:41.950200 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:41.950165    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:41.950200 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:41.950206    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:41.962962 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:41.962936    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:05:42.692631 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:05:42.692600    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:06:09.518503 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:09.518472    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"]
Jul 19 17:06:09.540578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:09.540542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:06:11.159766 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:11.159709    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"]
Jul 19 17:06:11.180014 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:11.179971    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:06:24.540682 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.540633    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz" podUID="6d86dd65-60d7-43e8-88fd-ef245dfa66c6" containerName="main" containerID="cri-o://b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3" gracePeriod=45
Jul 19 17:06:24.803511 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.803434    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:06:24.871710 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871671    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-home\") pod \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") "
Jul 19 17:06:24.871918 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-model-cache\") pod \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") "
Jul 19 17:06:24.871918 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871800    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knmnt\" (UniqueName: \"kubernetes.io/projected/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-kube-api-access-knmnt\") pod \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") "
Jul 19 17:06:24.871918 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871851    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tls-certs\") pod \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") "
Jul 19 17:06:24.871918 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871885    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-dshm\") pod \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") "
Jul 19 17:06:24.872129 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871922    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tmp-dir\") pod \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\" (UID: \"6d86dd65-60d7-43e8-88fd-ef245dfa66c6\") "
Jul 19 17:06:24.872129 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871977    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-home" (OuterVolumeSpecName: "home") pod "6d86dd65-60d7-43e8-88fd-ef245dfa66c6" (UID: "6d86dd65-60d7-43e8-88fd-ef245dfa66c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:24.872129 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.871987    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d86dd65-60d7-43e8-88fd-ef245dfa66c6" (UID: "6d86dd65-60d7-43e8-88fd-ef245dfa66c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:24.872418 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.872217    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:24.872418 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.872238    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:24.872418 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.872245    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d86dd65-60d7-43e8-88fd-ef245dfa66c6" (UID: "6d86dd65-60d7-43e8-88fd-ef245dfa66c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:24.874322 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.874300    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d86dd65-60d7-43e8-88fd-ef245dfa66c6" (UID: "6d86dd65-60d7-43e8-88fd-ef245dfa66c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:06:24.874580 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.874559    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-dshm" (OuterVolumeSpecName: "dshm") pod "6d86dd65-60d7-43e8-88fd-ef245dfa66c6" (UID: "6d86dd65-60d7-43e8-88fd-ef245dfa66c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:24.874646 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.874571    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-kube-api-access-knmnt" (OuterVolumeSpecName: "kube-api-access-knmnt") pod "6d86dd65-60d7-43e8-88fd-ef245dfa66c6" (UID: "6d86dd65-60d7-43e8-88fd-ef245dfa66c6"). InnerVolumeSpecName "kube-api-access-knmnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:06:24.887012 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.886984    2584 generic.go:356] "Generic (PLEG): container finished" podID="6d86dd65-60d7-43e8-88fd-ef245dfa66c6" containerID="b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3" exitCode=0
Jul 19 17:06:24.887132 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.887029    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"
Jul 19 17:06:24.887132 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.887032    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz" event={"ID":"6d86dd65-60d7-43e8-88fd-ef245dfa66c6","Type":"ContainerDied","Data":"b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3"}
Jul 19 17:06:24.887132 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.887083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz" event={"ID":"6d86dd65-60d7-43e8-88fd-ef245dfa66c6","Type":"ContainerDied","Data":"4c08ddf441fdce5cbadcf3d001f2e486b831af1a7451fea2bfec0011857a48e2"}
Jul 19 17:06:24.887132 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.887107    2584 scope.go:117] "RemoveContainer" containerID="b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3"
Jul 19 17:06:24.899924 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.899902    2584 scope.go:117] "RemoveContainer" containerID="b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3"
Jul 19 17:06:24.900206 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:06:24.900187    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3\": container with ID starting with b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3 not found: ID does not exist" containerID="b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3"
Jul 19 17:06:24.900261 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.900215    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3"} err="failed to get container status \"b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3\": rpc error: code = NotFound desc = could not find container \"b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3\": container with ID starting with b57a680e93caeaf7153e419b6bd53c92d84e27e79008db16b711f22b752982b3 not found: ID does not exist"
Jul 19 17:06:24.909883 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.909857    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"]
Jul 19 17:06:24.914634 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.914614    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-000f0eba-kserve-5999f65b9f-nzjdz"]
Jul 19 17:06:24.973729 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.973694    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:24.973729 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.973726    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:24.973729 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.973737    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knmnt\" (UniqueName: \"kubernetes.io/projected/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-kube-api-access-knmnt\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:24.973956 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:24.973745    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d86dd65-60d7-43e8-88fd-ef245dfa66c6-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.183513 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.183473    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4" podUID="a92ae97a-7755-4538-819a-f4c180932362" containerName="main" containerID="cri-o://e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da" gracePeriod=45
Jul 19 17:06:26.436619 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.436555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:06:26.486948 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.486917    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-home\") pod \"a92ae97a-7755-4538-819a-f4c180932362\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") "
Jul 19 17:06:26.487139 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.486973    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtgmf\" (UniqueName: \"kubernetes.io/projected/a92ae97a-7755-4538-819a-f4c180932362-kube-api-access-wtgmf\") pod \"a92ae97a-7755-4538-819a-f4c180932362\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") "
Jul 19 17:06:26.487139 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487005    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-tmp-dir\") pod \"a92ae97a-7755-4538-819a-f4c180932362\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") "
Jul 19 17:06:26.487139 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-dshm\") pod \"a92ae97a-7755-4538-819a-f4c180932362\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") "
Jul 19 17:06:26.487139 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487040    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-model-cache\") pod \"a92ae97a-7755-4538-819a-f4c180932362\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") "
Jul 19 17:06:26.487139 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487128    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a92ae97a-7755-4538-819a-f4c180932362-tls-certs\") pod \"a92ae97a-7755-4538-819a-f4c180932362\" (UID: \"a92ae97a-7755-4538-819a-f4c180932362\") "
Jul 19 17:06:26.487447 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487265    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-home" (OuterVolumeSpecName: "home") pod "a92ae97a-7755-4538-819a-f4c180932362" (UID: "a92ae97a-7755-4538-819a-f4c180932362"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:26.487447 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a92ae97a-7755-4538-819a-f4c180932362" (UID: "a92ae97a-7755-4538-819a-f4c180932362"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:26.487528 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487453    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-model-cache" (OuterVolumeSpecName: "model-cache") pod "a92ae97a-7755-4538-819a-f4c180932362" (UID: "a92ae97a-7755-4538-819a-f4c180932362"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:26.487528 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487472    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.487528 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.487489    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.489352 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.489323    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-dshm" (OuterVolumeSpecName: "dshm") pod "a92ae97a-7755-4538-819a-f4c180932362" (UID: "a92ae97a-7755-4538-819a-f4c180932362"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:06:26.489474 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.489345    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92ae97a-7755-4538-819a-f4c180932362-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a92ae97a-7755-4538-819a-f4c180932362" (UID: "a92ae97a-7755-4538-819a-f4c180932362"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:06:26.489474 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.489397    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92ae97a-7755-4538-819a-f4c180932362-kube-api-access-wtgmf" (OuterVolumeSpecName: "kube-api-access-wtgmf") pod "a92ae97a-7755-4538-819a-f4c180932362" (UID: "a92ae97a-7755-4538-819a-f4c180932362"). InnerVolumeSpecName "kube-api-access-wtgmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:06:26.587936 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.587899    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgmf\" (UniqueName: \"kubernetes.io/projected/a92ae97a-7755-4538-819a-f4c180932362-kube-api-access-wtgmf\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.587936 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.587928    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.587936 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.587937    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a92ae97a-7755-4538-819a-f4c180932362-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.587936 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.587947    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a92ae97a-7755-4538-819a-f4c180932362-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:06:26.632630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.632586    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d86dd65-60d7-43e8-88fd-ef245dfa66c6" path="/var/lib/kubelet/pods/6d86dd65-60d7-43e8-88fd-ef245dfa66c6/volumes"
Jul 19 17:06:26.898419 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.898338    2584 generic.go:356] "Generic (PLEG): container finished" podID="a92ae97a-7755-4538-819a-f4c180932362" containerID="e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da" exitCode=0
Jul 19 17:06:26.898419 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.898393    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"
Jul 19 17:06:26.898419 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.898408    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4" event={"ID":"a92ae97a-7755-4538-819a-f4c180932362","Type":"ContainerDied","Data":"e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da"}
Jul 19 17:06:26.898717 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.898439    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4" event={"ID":"a92ae97a-7755-4538-819a-f4c180932362","Type":"ContainerDied","Data":"0bc41d314b3193b8ec8ad37446caf9b3dadf44dceb986785a3c8d957156f9c4f"}
Jul 19 17:06:26.898717 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.898455    2584 scope.go:117] "RemoveContainer" containerID="e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da"
Jul 19 17:06:26.907555 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.907535    2584 scope.go:117] "RemoveContainer" containerID="e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da"
Jul 19 17:06:26.907847 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:06:26.907822    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da\": container with ID starting with e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da not found: ID does not exist" containerID="e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da"
Jul 19 17:06:26.907950 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.907853    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da"} err="failed to get container status \"e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da\": rpc error: code = NotFound desc = could not find container \"e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da\": container with ID starting with e36c4a314a05e2fc18fa06f53f03bccaa56af843ddbc4999c5eedadb707728da not found: ID does not exist"
Jul 19 17:06:26.913462 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.913440    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"]
Jul 19 17:06:26.918411 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:26.918387    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-000f0eba-kserve-6456f55f-66hm4"]
Jul 19 17:06:28.632116 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:06:28.632081    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92ae97a-7755-4538-819a-f4c180932362" path="/var/lib/kubelet/pods/a92ae97a-7755-4538-819a-f4c180932362/volumes"
Jul 19 17:08:10.668982 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:08:10.668956    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:08:10.673949 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:08:10.673928    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:09:18.064185 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:18.064148    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:27.594796 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:27.594763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:35.866258 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:35.866171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:40.965122 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:40.965087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:43.889633 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:43.889593    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:43.989811 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:43.989773    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:44.326089 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:44.326056    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:49.937548 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:49.937510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:50.038026 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:50.037987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:50.172152 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:50.172116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:50.470125 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:50.470092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:50.678339 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:50.678299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:53.044980 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.044935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:53.145315 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.145261    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:53.246979 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.246946    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"]
Jul 19 17:09:53.247390 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.247373    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92ae97a-7755-4538-819a-f4c180932362" containerName="main"
Jul 19 17:09:53.247390 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.247391    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92ae97a-7755-4538-819a-f4c180932362" containerName="main"
Jul 19 17:09:53.247533 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.247412    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d86dd65-60d7-43e8-88fd-ef245dfa66c6" containerName="main"
Jul 19 17:09:53.247533 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.247418    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d86dd65-60d7-43e8-88fd-ef245dfa66c6" containerName="main"
Jul 19 17:09:53.247533 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.247476    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92ae97a-7755-4538-819a-f4c180932362" containerName="main"
Jul 19 17:09:53.247533 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.247484    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d86dd65-60d7-43e8-88fd-ef245dfa66c6" containerName="main"
Jul 19 17:09:53.250393 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.250371    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.252768 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.252745    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-debb71c1-serving-cert\""
Jul 19 17:09:53.260340 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.260312    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"]
Jul 19 17:09:53.277366 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.277333    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:53.313881 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.313771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgwv\" (UniqueName: \"kubernetes.io/projected/b835a3e4-f4a6-45ea-a359-57ebba8975bf-kube-api-access-fzgwv\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.313881 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.313843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.314088 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.313905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b835a3e4-f4a6-45ea-a359-57ebba8975bf-openshift-service-ca\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.415155 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.415117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b835a3e4-f4a6-45ea-a359-57ebba8975bf-openshift-service-ca\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.415400 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.415213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgwv\" (UniqueName: \"kubernetes.io/projected/b835a3e4-f4a6-45ea-a359-57ebba8975bf-kube-api-access-fzgwv\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.415400 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.415270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.415541 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:09:53.415416    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-serving-cert: secret "maas-api-e2e-ait-debb71c1-serving-cert" not found
Jul 19 17:09:53.415541 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:09:53.415488    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls podName:b835a3e4-f4a6-45ea-a359-57ebba8975bf nodeName:}" failed. No retries permitted until 2026-07-19 17:09:53.915461975 +0000 UTC m=+1903.820713378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls") pod "maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" (UID: "b835a3e4-f4a6-45ea-a359-57ebba8975bf") : secret "maas-api-e2e-ait-debb71c1-serving-cert" not found
Jul 19 17:09:53.415773 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.415750    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b835a3e4-f4a6-45ea-a359-57ebba8975bf-openshift-service-ca\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.439757 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.439724    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgwv\" (UniqueName: \"kubernetes.io/projected/b835a3e4-f4a6-45ea-a359-57ebba8975bf-kube-api-access-fzgwv\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.624987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.624882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:53.909466 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.909431    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:53.920721 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.920688    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:53.923437 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:53.923410    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls\") pod \"maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:54.003587 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:54.003544    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:09:54.162139 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:54.162047    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:54.310568 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:54.310539    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"]
Jul 19 17:09:54.312694 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:09:54.312661    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb835a3e4_f4a6_45ea_a359_57ebba8975bf.slice/crio-74274f3330885c6eda96abde33f8c387b662774199d3902c08b47463d8d53b66 WatchSource:0}: Error finding container 74274f3330885c6eda96abde33f8c387b662774199d3902c08b47463d8d53b66: Status 404 returned error can't find the container with id 74274f3330885c6eda96abde33f8c387b662774199d3902c08b47463d8d53b66
Jul 19 17:09:54.314398 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:54.314372    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 17:09:54.899593 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:54.899567    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" event={"ID":"b835a3e4-f4a6-45ea-a359-57ebba8975bf","Type":"ContainerStarted","Data":"74274f3330885c6eda96abde33f8c387b662774199d3902c08b47463d8d53b66"}
Jul 19 17:09:55.905587 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:55.905549    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" event={"ID":"b835a3e4-f4a6-45ea-a359-57ebba8975bf","Type":"ContainerStarted","Data":"64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b"}
Jul 19 17:09:55.906016 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:55.905682    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:09:55.921562 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:09:55.921514    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" podStartSLOduration=2.402711963 podStartE2EDuration="2.921499518s" totalImagesPullingTime="518.787555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:09:55.920150091 +0000 UTC m=+1905.825401515" watchObservedRunningTime="2026-07-19 17:09:55.921499518 +0000 UTC m=+1905.826750941"
Jul 19 17:10:01.915737 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:01.915703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:10:18.604175 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.604136    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"]
Jul 19 17:10:18.607958 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.607939    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:18.610301 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.610261    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:10:18.613698 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.613673    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"]
Jul 19 17:10:18.744291 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.744239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138375f-0143-4326-9ac9-54b3f57b87e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-debb71c1-6wchh\" (UID: \"4138375f-0143-4326-9ac9-54b3f57b87e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:18.844956 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.844915    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138375f-0143-4326-9ac9-54b3f57b87e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-debb71c1-6wchh\" (UID: \"4138375f-0143-4326-9ac9-54b3f57b87e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:18.845651 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.845630    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138375f-0143-4326-9ac9-54b3f57b87e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-debb71c1-6wchh\" (UID: \"4138375f-0143-4326-9ac9-54b3f57b87e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:18.923865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:18.923832    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:19.070913 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:19.070889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"]
Jul 19 17:10:19.072800 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:10:19.072770    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4138375f_0143_4326_9ac9_54b3f57b87e9.slice/crio-c2b3e48ec43927a974a2199d20df31af41dd25c039ed8c2d09087922f9076bf0 WatchSource:0}: Error finding container c2b3e48ec43927a974a2199d20df31af41dd25c039ed8c2d09087922f9076bf0: Status 404 returned error can't find the container with id c2b3e48ec43927a974a2199d20df31af41dd25c039ed8c2d09087922f9076bf0
Jul 19 17:10:20.027332 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:20.027290    2584 generic.go:356] "Generic (PLEG): container finished" podID="4138375f-0143-4326-9ac9-54b3f57b87e9" containerID="37a4ca1c71251375f21e2e1a6d53f463ddffdffb014d333bc15bf71757cef7df" exitCode=0
Jul 19 17:10:20.027763 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:20.027393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh" event={"ID":"4138375f-0143-4326-9ac9-54b3f57b87e9","Type":"ContainerDied","Data":"37a4ca1c71251375f21e2e1a6d53f463ddffdffb014d333bc15bf71757cef7df"}
Jul 19 17:10:20.027763 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:20.027464    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh" event={"ID":"4138375f-0143-4326-9ac9-54b3f57b87e9","Type":"ContainerStarted","Data":"c2b3e48ec43927a974a2199d20df31af41dd25c039ed8c2d09087922f9076bf0"}
Jul 19 17:10:21.174299 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:21.174250    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:21.264435 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:21.264389    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138375f-0143-4326-9ac9-54b3f57b87e9-maas-api-service-ca\") pod \"4138375f-0143-4326-9ac9-54b3f57b87e9\" (UID: \"4138375f-0143-4326-9ac9-54b3f57b87e9\") "
Jul 19 17:10:21.264763 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:21.264739    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4138375f-0143-4326-9ac9-54b3f57b87e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4138375f-0143-4326-9ac9-54b3f57b87e9" (UID: "4138375f-0143-4326-9ac9-54b3f57b87e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:10:21.366043 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:21.365957    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4138375f-0143-4326-9ac9-54b3f57b87e9-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:10:22.038830 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:22.038797    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"
Jul 19 17:10:22.038830 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:22.038804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh" event={"ID":"4138375f-0143-4326-9ac9-54b3f57b87e9","Type":"ContainerDied","Data":"c2b3e48ec43927a974a2199d20df31af41dd25c039ed8c2d09087922f9076bf0"}
Jul 19 17:10:22.039097 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:22.038850    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2b3e48ec43927a974a2199d20df31af41dd25c039ed8c2d09087922f9076bf0"
Jul 19 17:10:28.978907 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:28.978870    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"]
Jul 19 17:10:28.979403 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:28.979131    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" podUID="b835a3e4-f4a6-45ea-a359-57ebba8975bf" containerName="maas-api" containerID="cri-o://64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b" gracePeriod=30
Jul 19 17:10:29.012653 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.012622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:29.238352 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.238260    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:10:29.332185 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.332145    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzgwv\" (UniqueName: \"kubernetes.io/projected/b835a3e4-f4a6-45ea-a359-57ebba8975bf-kube-api-access-fzgwv\") pod \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") "
Jul 19 17:10:29.332408 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.332193    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls\") pod \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") "
Jul 19 17:10:29.332408 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.332310    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b835a3e4-f4a6-45ea-a359-57ebba8975bf-openshift-service-ca\") pod \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\" (UID: \"b835a3e4-f4a6-45ea-a359-57ebba8975bf\") "
Jul 19 17:10:29.332683 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.332660    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b835a3e4-f4a6-45ea-a359-57ebba8975bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b835a3e4-f4a6-45ea-a359-57ebba8975bf" (UID: "b835a3e4-f4a6-45ea-a359-57ebba8975bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:10:29.334592 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.334561    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b835a3e4-f4a6-45ea-a359-57ebba8975bf-kube-api-access-fzgwv" (OuterVolumeSpecName: "kube-api-access-fzgwv") pod "b835a3e4-f4a6-45ea-a359-57ebba8975bf" (UID: "b835a3e4-f4a6-45ea-a359-57ebba8975bf"). InnerVolumeSpecName "kube-api-access-fzgwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:10:29.334592 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.334578    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b835a3e4-f4a6-45ea-a359-57ebba8975bf" (UID: "b835a3e4-f4a6-45ea-a359-57ebba8975bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:10:29.433808 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.433769    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgwv\" (UniqueName: \"kubernetes.io/projected/b835a3e4-f4a6-45ea-a359-57ebba8975bf-kube-api-access-fzgwv\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:10:29.433808 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.433805    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b835a3e4-f4a6-45ea-a359-57ebba8975bf-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:10:29.434017 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:29.433822    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b835a3e4-f4a6-45ea-a359-57ebba8975bf-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:10:30.079666 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.079629    2584 generic.go:356] "Generic (PLEG): container finished" podID="b835a3e4-f4a6-45ea-a359-57ebba8975bf" containerID="64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b" exitCode=0
Jul 19 17:10:30.080090 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.079672    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"
Jul 19 17:10:30.080090 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.079716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" event={"ID":"b835a3e4-f4a6-45ea-a359-57ebba8975bf","Type":"ContainerDied","Data":"64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b"}
Jul 19 17:10:30.080090 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.079772    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl" event={"ID":"b835a3e4-f4a6-45ea-a359-57ebba8975bf","Type":"ContainerDied","Data":"74274f3330885c6eda96abde33f8c387b662774199d3902c08b47463d8d53b66"}
Jul 19 17:10:30.080090 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.079796    2584 scope.go:117] "RemoveContainer" containerID="64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b"
Jul 19 17:10:30.090531 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.090509    2584 scope.go:117] "RemoveContainer" containerID="64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b"
Jul 19 17:10:30.090879 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:10:30.090860    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b\": container with ID starting with 64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b not found: ID does not exist" containerID="64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b"
Jul 19 17:10:30.090961 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.090889    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b"} err="failed to get container status \"64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b\": rpc error: code = NotFound desc = could not find container \"64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b\": container with ID starting with 64f4135f567520040e30737e4e19170ec949573c3553dba5d495290c26fecb2b not found: ID does not exist"
Jul 19 17:10:30.101857 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.101826    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"]
Jul 19 17:10:30.105199 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.105165    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-debb71c1-57d5579d4d-bjjhl"]
Jul 19 17:10:30.633052 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:30.633018    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b835a3e4-f4a6-45ea-a359-57ebba8975bf" path="/var/lib/kubelet/pods/b835a3e4-f4a6-45ea-a359-57ebba8975bf/volumes"
Jul 19 17:10:33.673260 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:33.673225    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"]
Jul 19 17:10:33.677216 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:33.677189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:33.677216 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:33.677208    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-debb71c1-6wchh"]
Jul 19 17:10:33.831273 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:33.831243    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:34.066326 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:34.066224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:34.335536 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:34.335448    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:34.428829 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:34.428792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:34.535673 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:34.535636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:34.634025 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:34.633917    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4138375f-0143-4326-9ac9-54b3f57b87e9" path="/var/lib/kubelet/pods/4138375f-0143-4326-9ac9-54b3f57b87e9/volumes"
Jul 19 17:10:34.839954 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:34.839921    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:35.551869 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:35.551838    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:37.401262 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.401229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:37.618681 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.618646    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"]
Jul 19 17:10:37.619038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.619023    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b835a3e4-f4a6-45ea-a359-57ebba8975bf" containerName="maas-api"
Jul 19 17:10:37.619038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.619039    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b835a3e4-f4a6-45ea-a359-57ebba8975bf" containerName="maas-api"
Jul 19 17:10:37.619190 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.619074    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4138375f-0143-4326-9ac9-54b3f57b87e9" containerName="revoke-keys"
Jul 19 17:10:37.619190 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.619084    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4138375f-0143-4326-9ac9-54b3f57b87e9" containerName="revoke-keys"
Jul 19 17:10:37.619190 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.619175    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b835a3e4-f4a6-45ea-a359-57ebba8975bf" containerName="maas-api"
Jul 19 17:10:37.619376 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.619188    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4138375f-0143-4326-9ac9-54b3f57b87e9" containerName="revoke-keys"
Jul 19 17:10:37.623699 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.623678    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.626074 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.626050    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5f384e8f-serving-cert\""
Jul 19 17:10:37.630155 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.630092    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"]
Jul 19 17:10:37.646363 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.646334    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:37.706928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.706848    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-maas-api-tls\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.707085 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.706975    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjklr\" (UniqueName: \"kubernetes.io/projected/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-kube-api-access-qjklr\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.707085 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.707005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.808327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.808274    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjklr\" (UniqueName: \"kubernetes.io/projected/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-kube-api-access-qjklr\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.808515 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.808340    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.808515 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.808412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-maas-api-tls\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.809108 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.809089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.811195 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.811167    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-maas-api-tls\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.816078 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.816057    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjklr\" (UniqueName: \"kubernetes.io/projected/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-kube-api-access-qjklr\") pod \"maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:37.840595 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.840570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:37.936350 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:37.936310    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:38.083308 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:38.083215    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"]
Jul 19 17:10:38.086226 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:10:38.086200    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3cd5be_1210_4f52_96d3_f7a9e46092d0.slice/crio-4ec3240b5b7f1b36ddb402aeae5173095fc31fd36b26ff5ab3cc3b068d1a0b68 WatchSource:0}: Error finding container 4ec3240b5b7f1b36ddb402aeae5173095fc31fd36b26ff5ab3cc3b068d1a0b68: Status 404 returned error can't find the container with id 4ec3240b5b7f1b36ddb402aeae5173095fc31fd36b26ff5ab3cc3b068d1a0b68
Jul 19 17:10:38.121007 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:38.120959    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz" event={"ID":"fd3cd5be-1210-4f52-96d3-f7a9e46092d0","Type":"ContainerStarted","Data":"4ec3240b5b7f1b36ddb402aeae5173095fc31fd36b26ff5ab3cc3b068d1a0b68"}
Jul 19 17:10:38.176478 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:38.176444    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:38.283687 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:38.283588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:10:39.129888 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:39.129849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz" event={"ID":"fd3cd5be-1210-4f52-96d3-f7a9e46092d0","Type":"ContainerStarted","Data":"ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d"}
Jul 19 17:10:39.130327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:39.129971    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:10:39.146774 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:39.146718    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz" podStartSLOduration=1.871892855 podStartE2EDuration="2.146702031s" totalImagesPullingTime="274.809176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:10:39.145271478 +0000 UTC m=+1949.050522902" watchObservedRunningTime="2026-07-19 17:10:39.146702031 +0000 UTC m=+1949.051953455"
Jul 19 17:10:45.140996 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:10:45.140961    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:11:02.421327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.421233    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"]
Jul 19 17:11:02.424780 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.424762    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:02.427061 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.427040    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:11:02.433090 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.432984    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"]
Jul 19 17:11:02.527256 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.527219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037362ea-40ca-4f56-b263-1cc6459b163a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl\" (UID: \"037362ea-40ca-4f56-b263-1cc6459b163a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:02.629476 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.628537    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037362ea-40ca-4f56-b263-1cc6459b163a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl\" (UID: \"037362ea-40ca-4f56-b263-1cc6459b163a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:02.629476 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.629427    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037362ea-40ca-4f56-b263-1cc6459b163a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl\" (UID: \"037362ea-40ca-4f56-b263-1cc6459b163a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:02.739653 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.739561    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:02.884319 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:02.884272    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"]
Jul 19 17:11:02.886156 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:11:02.886121    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037362ea_40ca_4f56_b263_1cc6459b163a.slice/crio-0b461762dac846e04473489170e2b14a7ea960e5c72e6cfcb20dcf31710e75b1 WatchSource:0}: Error finding container 0b461762dac846e04473489170e2b14a7ea960e5c72e6cfcb20dcf31710e75b1: Status 404 returned error can't find the container with id 0b461762dac846e04473489170e2b14a7ea960e5c72e6cfcb20dcf31710e75b1
Jul 19 17:11:03.246906 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:03.246865    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl" event={"ID":"037362ea-40ca-4f56-b263-1cc6459b163a","Type":"ContainerStarted","Data":"e553a9d122dcc8e92cc7599945fb1c58972605dac44cd27fd8ff5499cd5b6807"}
Jul 19 17:11:03.247115 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:03.246931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl" event={"ID":"037362ea-40ca-4f56-b263-1cc6459b163a","Type":"ContainerStarted","Data":"0b461762dac846e04473489170e2b14a7ea960e5c72e6cfcb20dcf31710e75b1"}
Jul 19 17:11:03.263659 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:03.263610    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl" podStartSLOduration=1.263593584 podStartE2EDuration="1.263593584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:11:03.261130007 +0000 UTC m=+1973.166381433" watchObservedRunningTime="2026-07-19 17:11:03.263593584 +0000 UTC m=+1973.168845021"
Jul 19 17:11:04.253344 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:04.253272    2584 generic.go:356] "Generic (PLEG): container finished" podID="037362ea-40ca-4f56-b263-1cc6459b163a" containerID="e553a9d122dcc8e92cc7599945fb1c58972605dac44cd27fd8ff5499cd5b6807" exitCode=0
Jul 19 17:11:04.253719 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:04.253380    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl" event={"ID":"037362ea-40ca-4f56-b263-1cc6459b163a","Type":"ContainerDied","Data":"e553a9d122dcc8e92cc7599945fb1c58972605dac44cd27fd8ff5499cd5b6807"}
Jul 19 17:11:05.394777 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:05.394747    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:05.554001 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:05.553911    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037362ea-40ca-4f56-b263-1cc6459b163a-maas-api-service-ca\") pod \"037362ea-40ca-4f56-b263-1cc6459b163a\" (UID: \"037362ea-40ca-4f56-b263-1cc6459b163a\") "
Jul 19 17:11:05.554230 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:05.554205    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037362ea-40ca-4f56-b263-1cc6459b163a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "037362ea-40ca-4f56-b263-1cc6459b163a" (UID: "037362ea-40ca-4f56-b263-1cc6459b163a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:11:05.655447 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:05.655414    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/037362ea-40ca-4f56-b263-1cc6459b163a-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:06.264777 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:06.264740    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl" event={"ID":"037362ea-40ca-4f56-b263-1cc6459b163a","Type":"ContainerDied","Data":"0b461762dac846e04473489170e2b14a7ea960e5c72e6cfcb20dcf31710e75b1"}
Jul 19 17:11:06.264777 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:06.264760    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"
Jul 19 17:11:06.264978 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:06.264796    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b461762dac846e04473489170e2b14a7ea960e5c72e6cfcb20dcf31710e75b1"
Jul 19 17:11:12.919973 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:12.919932    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"]
Jul 19 17:11:12.920472 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:12.920176    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz" podUID="fd3cd5be-1210-4f52-96d3-f7a9e46092d0" containerName="maas-api" containerID="cri-o://ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d" gracePeriod=30
Jul 19 17:11:12.942031 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:12.942004    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:13.174429 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.174370    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:11:13.299533 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.299501    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd3cd5be-1210-4f52-96d3-f7a9e46092d0" containerID="ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d" exitCode=0
Jul 19 17:11:13.299533 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.299535    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"
Jul 19 17:11:13.299746 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.299589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz" event={"ID":"fd3cd5be-1210-4f52-96d3-f7a9e46092d0","Type":"ContainerDied","Data":"ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d"}
Jul 19 17:11:13.299746 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.299645    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz" event={"ID":"fd3cd5be-1210-4f52-96d3-f7a9e46092d0","Type":"ContainerDied","Data":"4ec3240b5b7f1b36ddb402aeae5173095fc31fd36b26ff5ab3cc3b068d1a0b68"}
Jul 19 17:11:13.299746 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.299668    2584 scope.go:117] "RemoveContainer" containerID="ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d"
Jul 19 17:11:13.308325 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.308303    2584 scope.go:117] "RemoveContainer" containerID="ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d"
Jul 19 17:11:13.308636 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:11:13.308595    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d\": container with ID starting with ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d not found: ID does not exist" containerID="ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d"
Jul 19 17:11:13.308636 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.308624    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d"} err="failed to get container status \"ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d\": rpc error: code = NotFound desc = could not find container \"ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d\": container with ID starting with ef5dd660b91a2a71996dd42862e3e7f8051a3f60bc28645d851d9954accb9e6d not found: ID does not exist"
Jul 19 17:11:13.323969 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.323941    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjklr\" (UniqueName: \"kubernetes.io/projected/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-kube-api-access-qjklr\") pod \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") "
Jul 19 17:11:13.324132 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.323996    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-maas-api-tls\") pod \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") "
Jul 19 17:11:13.324132 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.324026    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-openshift-service-ca\") pod \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\" (UID: \"fd3cd5be-1210-4f52-96d3-f7a9e46092d0\") "
Jul 19 17:11:13.324503 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.324481    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd3cd5be-1210-4f52-96d3-f7a9e46092d0" (UID: "fd3cd5be-1210-4f52-96d3-f7a9e46092d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:11:13.326347 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.326313    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-kube-api-access-qjklr" (OuterVolumeSpecName: "kube-api-access-qjklr") pod "fd3cd5be-1210-4f52-96d3-f7a9e46092d0" (UID: "fd3cd5be-1210-4f52-96d3-f7a9e46092d0"). InnerVolumeSpecName "kube-api-access-qjklr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:11:13.326461 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.326408    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd3cd5be-1210-4f52-96d3-f7a9e46092d0" (UID: "fd3cd5be-1210-4f52-96d3-f7a9e46092d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:11:13.425108 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.425020    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:13.425108 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.425048    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:13.425108 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.425058    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjklr\" (UniqueName: \"kubernetes.io/projected/fd3cd5be-1210-4f52-96d3-f7a9e46092d0-kube-api-access-qjklr\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:13.621131 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.621098    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"]
Jul 19 17:11:13.624711 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:13.624688    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f384e8f-8469ffbbbc-s26xz"]
Jul 19 17:11:14.632271 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:14.632237    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3cd5be-1210-4f52-96d3-f7a9e46092d0" path="/var/lib/kubelet/pods/fd3cd5be-1210-4f52-96d3-f7a9e46092d0/volumes"
Jul 19 17:11:17.494191 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:17.494150    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:17.494870 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:17.494534    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"]
Jul 19 17:11:17.496486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:17.496467    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f384e8f-g2rzl"]
Jul 19 17:11:17.649796 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:17.649766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:17.893744 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:17.893645    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:18.147530 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:18.147418    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:18.557484 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:18.557451    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:18.632458 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:18.632423    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037362ea-40ca-4f56-b263-1cc6459b163a" path="/var/lib/kubelet/pods/037362ea-40ca-4f56-b263-1cc6459b163a/volumes"
Jul 19 17:11:18.728188 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:18.728151    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:20.516052 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.515999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:20.615388 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.615347    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:20.641133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641104    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"]
Jul 19 17:11:20.641573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641554    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3cd5be-1210-4f52-96d3-f7a9e46092d0" containerName="maas-api"
Jul 19 17:11:20.641619 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641578    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3cd5be-1210-4f52-96d3-f7a9e46092d0" containerName="maas-api"
Jul 19 17:11:20.641655 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641622    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037362ea-40ca-4f56-b263-1cc6459b163a" containerName="revoke-keys"
Jul 19 17:11:20.641655 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641631    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="037362ea-40ca-4f56-b263-1cc6459b163a" containerName="revoke-keys"
Jul 19 17:11:20.641736 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641713    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3cd5be-1210-4f52-96d3-f7a9e46092d0" containerName="maas-api"
Jul 19 17:11:20.641736 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.641725    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="037362ea-40ca-4f56-b263-1cc6459b163a" containerName="revoke-keys"
Jul 19 17:11:20.644737 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.644721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.647232 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.647198    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-77bade34-serving-cert\""
Jul 19 17:11:20.656071 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.656048    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"]
Jul 19 17:11:20.684815 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.684778    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.685537 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.685511    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlzn\" (UniqueName: \"kubernetes.io/projected/a1befdf1-9c08-409b-a9a5-7f4d311f3874-kube-api-access-jjlzn\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.685778 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.685763    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1befdf1-9c08-409b-a9a5-7f4d311f3874-openshift-service-ca\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.715328 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.715290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:20.787742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.787660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.787742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.787704    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlzn\" (UniqueName: \"kubernetes.io/projected/a1befdf1-9c08-409b-a9a5-7f4d311f3874-kube-api-access-jjlzn\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.787978 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.787755    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1befdf1-9c08-409b-a9a5-7f4d311f3874-openshift-service-ca\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.787978 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:11:20.787825    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-serving-cert: secret "maas-api-e2e-ait-77bade34-serving-cert" not found
Jul 19 17:11:20.787978 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:11:20.787896    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls podName:a1befdf1-9c08-409b-a9a5-7f4d311f3874 nodeName:}" failed. No retries permitted until 2026-07-19 17:11:21.287874702 +0000 UTC m=+1991.193126120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls") pod "maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" (UID: "a1befdf1-9c08-409b-a9a5-7f4d311f3874") : secret "maas-api-e2e-ait-77bade34-serving-cert" not found
Jul 19 17:11:20.788446 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.788425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1befdf1-9c08-409b-a9a5-7f4d311f3874-openshift-service-ca\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.798573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.798548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlzn\" (UniqueName: \"kubernetes.io/projected/a1befdf1-9c08-409b-a9a5-7f4d311f3874-kube-api-access-jjlzn\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:20.985879 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:20.985811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:21.201926 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:21.201892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:21.293442 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:21.293403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:21.295930 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:21.295905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls\") pod \"maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:21.302609 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:21.302577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:21.557317 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:21.557211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:21.706178 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:21.706153    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"]
Jul 19 17:11:21.708368 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:11:21.708337    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1befdf1_9c08_409b_a9a5_7f4d311f3874.slice/crio-9165d061327594f17bc5ff577edf2e031bf94e0f1a6a958f95cc38980b8ebcf1 WatchSource:0}: Error finding container 9165d061327594f17bc5ff577edf2e031bf94e0f1a6a958f95cc38980b8ebcf1: Status 404 returned error can't find the container with id 9165d061327594f17bc5ff577edf2e031bf94e0f1a6a958f95cc38980b8ebcf1
Jul 19 17:11:22.346676 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:22.346572    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" event={"ID":"a1befdf1-9c08-409b-a9a5-7f4d311f3874","Type":"ContainerStarted","Data":"ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172"}
Jul 19 17:11:22.346676 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:22.346639    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:22.346676 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:22.346660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" event={"ID":"a1befdf1-9c08-409b-a9a5-7f4d311f3874","Type":"ContainerStarted","Data":"9165d061327594f17bc5ff577edf2e031bf94e0f1a6a958f95cc38980b8ebcf1"}
Jul 19 17:11:22.363595 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:22.363548    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" podStartSLOduration=2.000961352 podStartE2EDuration="2.363531247s" totalImagesPullingTime="362.569895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:11:22.362301767 +0000 UTC m=+1992.267553183" watchObservedRunningTime="2026-07-19 17:11:22.363531247 +0000 UTC m=+1992.268782711"
Jul 19 17:11:28.357979 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:28.357949    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:46.357390 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.357356    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"]
Jul 19 17:11:46.360718 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.360698    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:46.363161 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.363135    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:11:46.368073 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.368044    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"]
Jul 19 17:11:46.415360 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.415328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e86421b-b72d-4477-9af7-d1f34e417cc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-77bade34-8dcc9\" (UID: \"0e86421b-b72d-4477-9af7-d1f34e417cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:46.516737 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.516698    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e86421b-b72d-4477-9af7-d1f34e417cc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-77bade34-8dcc9\" (UID: \"0e86421b-b72d-4477-9af7-d1f34e417cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:46.517591 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.517568    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e86421b-b72d-4477-9af7-d1f34e417cc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-77bade34-8dcc9\" (UID: \"0e86421b-b72d-4477-9af7-d1f34e417cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:46.673558 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.673523    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:46.809505 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:46.809481    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"]
Jul 19 17:11:46.811697 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:11:46.811656    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e86421b_b72d_4477_9af7_d1f34e417cc6.slice/crio-6d4fa70af1c509198cb6bbf2d8f10435df85d887925abea3986b4b4707b56508 WatchSource:0}: Error finding container 6d4fa70af1c509198cb6bbf2d8f10435df85d887925abea3986b4b4707b56508: Status 404 returned error can't find the container with id 6d4fa70af1c509198cb6bbf2d8f10435df85d887925abea3986b4b4707b56508
Jul 19 17:11:47.478987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:47.478949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9" event={"ID":"0e86421b-b72d-4477-9af7-d1f34e417cc6","Type":"ContainerStarted","Data":"8cbd85d1c6db9f975268ae020752bf492fc1b1302ba0ad26e88b31c3ea9fc01b"}
Jul 19 17:11:47.479423 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:47.479015    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9" event={"ID":"0e86421b-b72d-4477-9af7-d1f34e417cc6","Type":"ContainerStarted","Data":"6d4fa70af1c509198cb6bbf2d8f10435df85d887925abea3986b4b4707b56508"}
Jul 19 17:11:47.493865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:47.493821    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9" podStartSLOduration=1.493803824 podStartE2EDuration="1.493803824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:11:47.492650294 +0000 UTC m=+2017.397901717" watchObservedRunningTime="2026-07-19 17:11:47.493803824 +0000 UTC m=+2017.399055283"
Jul 19 17:11:48.485378 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:48.485342    2584 generic.go:356] "Generic (PLEG): container finished" podID="0e86421b-b72d-4477-9af7-d1f34e417cc6" containerID="8cbd85d1c6db9f975268ae020752bf492fc1b1302ba0ad26e88b31c3ea9fc01b" exitCode=0
Jul 19 17:11:48.485841 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:48.485441    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9" event={"ID":"0e86421b-b72d-4477-9af7-d1f34e417cc6","Type":"ContainerDied","Data":"8cbd85d1c6db9f975268ae020752bf492fc1b1302ba0ad26e88b31c3ea9fc01b"}
Jul 19 17:11:49.632344 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:49.632314    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:49.743534 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:49.743419    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e86421b-b72d-4477-9af7-d1f34e417cc6-maas-api-service-ca\") pod \"0e86421b-b72d-4477-9af7-d1f34e417cc6\" (UID: \"0e86421b-b72d-4477-9af7-d1f34e417cc6\") "
Jul 19 17:11:49.743803 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:49.743773    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e86421b-b72d-4477-9af7-d1f34e417cc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e86421b-b72d-4477-9af7-d1f34e417cc6" (UID: "0e86421b-b72d-4477-9af7-d1f34e417cc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:11:49.844824 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:49.844797    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e86421b-b72d-4477-9af7-d1f34e417cc6-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:50.496995 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:50.496967    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"
Jul 19 17:11:50.497167 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:50.497008    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9" event={"ID":"0e86421b-b72d-4477-9af7-d1f34e417cc6","Type":"ContainerDied","Data":"6d4fa70af1c509198cb6bbf2d8f10435df85d887925abea3986b4b4707b56508"}
Jul 19 17:11:50.497167 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:50.497050    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4fa70af1c509198cb6bbf2d8f10435df85d887925abea3986b4b4707b56508"
Jul 19 17:11:56.810065 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:56.810027    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"]
Jul 19 17:11:56.810593 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:56.810272    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" podUID="a1befdf1-9c08-409b-a9a5-7f4d311f3874" containerName="maas-api" containerID="cri-o://ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172" gracePeriod=30
Jul 19 17:11:56.833439 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:56.833407    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:11:57.061692 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.061630    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:57.214469 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.214430    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjlzn\" (UniqueName: \"kubernetes.io/projected/a1befdf1-9c08-409b-a9a5-7f4d311f3874-kube-api-access-jjlzn\") pod \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") "
Jul 19 17:11:57.214659 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.214540    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls\") pod \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") "
Jul 19 17:11:57.214659 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.214577    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1befdf1-9c08-409b-a9a5-7f4d311f3874-openshift-service-ca\") pod \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\" (UID: \"a1befdf1-9c08-409b-a9a5-7f4d311f3874\") "
Jul 19 17:11:57.214988 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.214953    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1befdf1-9c08-409b-a9a5-7f4d311f3874-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1befdf1-9c08-409b-a9a5-7f4d311f3874" (UID: "a1befdf1-9c08-409b-a9a5-7f4d311f3874"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:11:57.216686 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.216655    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1befdf1-9c08-409b-a9a5-7f4d311f3874-kube-api-access-jjlzn" (OuterVolumeSpecName: "kube-api-access-jjlzn") pod "a1befdf1-9c08-409b-a9a5-7f4d311f3874" (UID: "a1befdf1-9c08-409b-a9a5-7f4d311f3874"). InnerVolumeSpecName "kube-api-access-jjlzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:11:57.216805 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.216707    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1befdf1-9c08-409b-a9a5-7f4d311f3874" (UID: "a1befdf1-9c08-409b-a9a5-7f4d311f3874"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:11:57.316256 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.316167    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlzn\" (UniqueName: \"kubernetes.io/projected/a1befdf1-9c08-409b-a9a5-7f4d311f3874-kube-api-access-jjlzn\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:57.316256 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.316195    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1befdf1-9c08-409b-a9a5-7f4d311f3874-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:57.316256 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.316204    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1befdf1-9c08-409b-a9a5-7f4d311f3874-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:11:57.533310 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.533251    2584 generic.go:356] "Generic (PLEG): container finished" podID="a1befdf1-9c08-409b-a9a5-7f4d311f3874" containerID="ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172" exitCode=0
Jul 19 17:11:57.533473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.533326    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"
Jul 19 17:11:57.533473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.533321    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" event={"ID":"a1befdf1-9c08-409b-a9a5-7f4d311f3874","Type":"ContainerDied","Data":"ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172"}
Jul 19 17:11:57.533473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.533392    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq" event={"ID":"a1befdf1-9c08-409b-a9a5-7f4d311f3874","Type":"ContainerDied","Data":"9165d061327594f17bc5ff577edf2e031bf94e0f1a6a958f95cc38980b8ebcf1"}
Jul 19 17:11:57.533473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.533423    2584 scope.go:117] "RemoveContainer" containerID="ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172"
Jul 19 17:11:57.542502 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.542484    2584 scope.go:117] "RemoveContainer" containerID="ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172"
Jul 19 17:11:57.542764 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:11:57.542748    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172\": container with ID starting with ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172 not found: ID does not exist" containerID="ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172"
Jul 19 17:11:57.542824 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.542776    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172"} err="failed to get container status \"ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172\": rpc error: code = NotFound desc = could not find container \"ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172\": container with ID starting with ad5b74a433e2b8d747980ccbeec115b82aafcb620636032bd6ff32eaf66c9172 not found: ID does not exist"
Jul 19 17:11:57.554410 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.554373    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"]
Jul 19 17:11:57.556097 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:57.556078    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77bade34-7cb5546d48-g7wmq"]
Jul 19 17:11:58.637985 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:11:58.637944    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1befdf1-9c08-409b-a9a5-7f4d311f3874" path="/var/lib/kubelet/pods/a1befdf1-9c08-409b-a9a5-7f4d311f3874/volumes"
Jul 19 17:12:01.432658 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:01.432617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:01.433142 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:01.433124    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"]
Jul 19 17:12:01.434786 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:01.434765    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-77bade34-8dcc9"]
Jul 19 17:12:02.635329 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:02.635127    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e86421b-b72d-4477-9af7-d1f34e417cc6" path="/var/lib/kubelet/pods/0e86421b-b72d-4477-9af7-d1f34e417cc6/volumes"
Jul 19 17:12:02.665105 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:02.665069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:02.881798 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:02.881763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:03.101040 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:03.101006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:03.412622 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:03.412590    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:03.650506 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:03.650471    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:05.820138 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.820093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:05.913983 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.913948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:05.919726 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.919699    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"]
Jul 19 17:12:05.920082 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.920070    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e86421b-b72d-4477-9af7-d1f34e417cc6" containerName="revoke-keys"
Jul 19 17:12:05.920125 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.920085    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e86421b-b72d-4477-9af7-d1f34e417cc6" containerName="revoke-keys"
Jul 19 17:12:05.920125 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.920098    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1befdf1-9c08-409b-a9a5-7f4d311f3874" containerName="maas-api"
Jul 19 17:12:05.920125 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.920104    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1befdf1-9c08-409b-a9a5-7f4d311f3874" containerName="maas-api"
Jul 19 17:12:05.920222 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.920168    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e86421b-b72d-4477-9af7-d1f34e417cc6" containerName="revoke-keys"
Jul 19 17:12:05.920222 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.920177    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1befdf1-9c08-409b-a9a5-7f4d311f3874" containerName="maas-api"
Jul 19 17:12:05.924510 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.924493    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:05.927133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.927112    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-352a195c-serving-cert\""
Jul 19 17:12:05.932372 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.932345    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"]
Jul 19 17:12:05.975064 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.975032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:05.996069 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.996034    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgzc\" (UniqueName: \"kubernetes.io/projected/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-kube-api-access-twgzc\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:05.996251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.996092    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-openshift-service-ca\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:05.996251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:05.996135    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.096665 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.096579    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgzc\" (UniqueName: \"kubernetes.io/projected/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-kube-api-access-twgzc\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.096665 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.096629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-openshift-service-ca\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.096665 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.096662    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.096926 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:12:06.096759    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-serving-cert: secret "maas-api-e2e-derive-352a195c-serving-cert" not found
Jul 19 17:12:06.096926 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:12:06.096824    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls podName:9ca682f6-7c31-47bb-ab9f-c8f088b55cfa nodeName:}" failed. No retries permitted until 2026-07-19 17:12:06.596807642 +0000 UTC m=+2036.502059044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls") pod "maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" (UID: "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa") : secret "maas-api-e2e-derive-352a195c-serving-cert" not found
Jul 19 17:12:06.097318 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.097300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-openshift-service-ca\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.107274 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.107250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgzc\" (UniqueName: \"kubernetes.io/projected/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-kube-api-access-twgzc\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.200156 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.200127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:06.455528 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.455497    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:06.556844 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.556806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:06.599817 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.599783    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.602437 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.602407    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls\") pod \"maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.837935 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.837835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:06.979584 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:06.979523    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"]
Jul 19 17:12:06.982907 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:12:06.982863    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca682f6_7c31_47bb_ab9f_c8f088b55cfa.slice/crio-62e88daf73b14f24081399d4c201f27962a5f3a04ce8c45f40d75e9d854212c3 WatchSource:0}: Error finding container 62e88daf73b14f24081399d4c201f27962a5f3a04ce8c45f40d75e9d854212c3: Status 404 returned error can't find the container with id 62e88daf73b14f24081399d4c201f27962a5f3a04ce8c45f40d75e9d854212c3
Jul 19 17:12:07.585315 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:07.585256    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" event={"ID":"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa","Type":"ContainerStarted","Data":"d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60"}
Jul 19 17:12:07.585819 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:07.585782    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" event={"ID":"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa","Type":"ContainerStarted","Data":"62e88daf73b14f24081399d4c201f27962a5f3a04ce8c45f40d75e9d854212c3"}
Jul 19 17:12:07.585927 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:07.585827    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:07.603556 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:07.603468    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" podStartSLOduration=2.217867533 podStartE2EDuration="2.603416434s" totalImagesPullingTime="385.548901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:12:07.600756001 +0000 UTC m=+2037.506007440" watchObservedRunningTime="2026-07-19 17:12:07.603416434 +0000 UTC m=+2037.508667857"
Jul 19 17:12:14.602089 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:14.602061    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:25.912828 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:25.912774    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:26.017164 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:26.017130    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:27.262429 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:27.262397    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:27.362849 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:27.362818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:30.620512 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.620474    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"]
Jul 19 17:12:30.623943 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.623926    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:30.626537 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.626514    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:12:30.635405 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.635378    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"]
Jul 19 17:12:30.707692 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.707659    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a831a-b794-4a85-b78e-9428fbe2b5a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-352a195c-llfz2\" (UID: \"380a831a-b794-4a85-b78e-9428fbe2b5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:30.809007 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.808969    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a831a-b794-4a85-b78e-9428fbe2b5a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-352a195c-llfz2\" (UID: \"380a831a-b794-4a85-b78e-9428fbe2b5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:30.809677 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.809656    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a831a-b794-4a85-b78e-9428fbe2b5a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-352a195c-llfz2\" (UID: \"380a831a-b794-4a85-b78e-9428fbe2b5a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:30.937619 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:30.937580    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:31.080454 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:31.080424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"]
Jul 19 17:12:31.083698 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:12:31.083654    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380a831a_b794_4a85_b78e_9428fbe2b5a8.slice/crio-a2915c44f795a16f2f7ac23011b1411dc8eef0f06c95a5c7d0a738154b0ab468 WatchSource:0}: Error finding container a2915c44f795a16f2f7ac23011b1411dc8eef0f06c95a5c7d0a738154b0ab468: Status 404 returned error can't find the container with id a2915c44f795a16f2f7ac23011b1411dc8eef0f06c95a5c7d0a738154b0ab468
Jul 19 17:12:31.708079 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:31.708035    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2" event={"ID":"380a831a-b794-4a85-b78e-9428fbe2b5a8","Type":"ContainerStarted","Data":"d7f8e3724930b3d24a498b9276e885172e872c942d54dc4f71c9e4700e06221b"}
Jul 19 17:12:31.708578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:31.708093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2" event={"ID":"380a831a-b794-4a85-b78e-9428fbe2b5a8","Type":"ContainerStarted","Data":"a2915c44f795a16f2f7ac23011b1411dc8eef0f06c95a5c7d0a738154b0ab468"}
Jul 19 17:12:31.722803 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:31.722735    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2" podStartSLOduration=1.7227210020000001 podStartE2EDuration="1.722721002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:12:31.721560241 +0000 UTC m=+2061.626811665" watchObservedRunningTime="2026-07-19 17:12:31.722721002 +0000 UTC m=+2061.627972425"
Jul 19 17:12:32.034671 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:32.034578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:32.714300 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:32.714246    2584 generic.go:356] "Generic (PLEG): container finished" podID="380a831a-b794-4a85-b78e-9428fbe2b5a8" containerID="d7f8e3724930b3d24a498b9276e885172e872c942d54dc4f71c9e4700e06221b" exitCode=0
Jul 19 17:12:32.714671 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:32.714338    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2" event={"ID":"380a831a-b794-4a85-b78e-9428fbe2b5a8","Type":"ContainerDied","Data":"d7f8e3724930b3d24a498b9276e885172e872c942d54dc4f71c9e4700e06221b"}
Jul 19 17:12:33.858713 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:33.858680    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:33.935039 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:33.935000    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a831a-b794-4a85-b78e-9428fbe2b5a8-maas-api-service-ca\") pod \"380a831a-b794-4a85-b78e-9428fbe2b5a8\" (UID: \"380a831a-b794-4a85-b78e-9428fbe2b5a8\") "
Jul 19 17:12:33.935419 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:33.935396    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380a831a-b794-4a85-b78e-9428fbe2b5a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "380a831a-b794-4a85-b78e-9428fbe2b5a8" (UID: "380a831a-b794-4a85-b78e-9428fbe2b5a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:12:34.036555 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:34.036471    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/380a831a-b794-4a85-b78e-9428fbe2b5a8-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:12:34.725045 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:34.724986    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"
Jul 19 17:12:34.725045 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:34.725031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2" event={"ID":"380a831a-b794-4a85-b78e-9428fbe2b5a8","Type":"ContainerDied","Data":"a2915c44f795a16f2f7ac23011b1411dc8eef0f06c95a5c7d0a738154b0ab468"}
Jul 19 17:12:34.725261 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:34.725085    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2915c44f795a16f2f7ac23011b1411dc8eef0f06c95a5c7d0a738154b0ab468"
Jul 19 17:12:41.006053 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.006010    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"]
Jul 19 17:12:41.006572 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.006268    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" podUID="9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" containerName="maas-api" containerID="cri-o://d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60" gracePeriod=30
Jul 19 17:12:41.031887 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.031861    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:41.266977 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.266910    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:41.406544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.406507    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twgzc\" (UniqueName: \"kubernetes.io/projected/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-kube-api-access-twgzc\") pod \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") "
Jul 19 17:12:41.406751 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.406563    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls\") pod \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") "
Jul 19 17:12:41.406751 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.406640    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-openshift-service-ca\") pod \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\" (UID: \"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa\") "
Jul 19 17:12:41.407022 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.406988    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" (UID: "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:12:41.408933 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.408909    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-kube-api-access-twgzc" (OuterVolumeSpecName: "kube-api-access-twgzc") pod "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" (UID: "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa"). InnerVolumeSpecName "kube-api-access-twgzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:12:41.409066 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.409051    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" (UID: "9ca682f6-7c31-47bb-ab9f-c8f088b55cfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:12:41.507906 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.507862    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:12:41.507906 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.507900    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgzc\" (UniqueName: \"kubernetes.io/projected/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-kube-api-access-twgzc\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:12:41.507906 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.507913    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:12:41.762071 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.762039    2584 generic.go:356] "Generic (PLEG): container finished" podID="9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" containerID="d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60" exitCode=0
Jul 19 17:12:41.762257 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.762081    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"
Jul 19 17:12:41.762257 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.762085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" event={"ID":"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa","Type":"ContainerDied","Data":"d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60"}
Jul 19 17:12:41.762257 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.762130    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25" event={"ID":"9ca682f6-7c31-47bb-ab9f-c8f088b55cfa","Type":"ContainerDied","Data":"62e88daf73b14f24081399d4c201f27962a5f3a04ce8c45f40d75e9d854212c3"}
Jul 19 17:12:41.762257 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.762151    2584 scope.go:117] "RemoveContainer" containerID="d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60"
Jul 19 17:12:41.771864 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.771839    2584 scope.go:117] "RemoveContainer" containerID="d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60"
Jul 19 17:12:41.772183 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:12:41.772164    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60\": container with ID starting with d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60 not found: ID does not exist" containerID="d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60"
Jul 19 17:12:41.772246 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.772191    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60"} err="failed to get container status \"d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60\": rpc error: code = NotFound desc = could not find container \"d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60\": container with ID starting with d8ca5281bdff0d05ae79ed40c4e3ecff94d37f1c5de54ecf915c0c57a1fc8e60 not found: ID does not exist"
Jul 19 17:12:41.782330 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.782298    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"]
Jul 19 17:12:41.786736 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:41.786706    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-352a195c-6bddcc48b7-vrs25"]
Jul 19 17:12:42.632761 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:42.632726    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" path="/var/lib/kubelet/pods/9ca682f6-7c31-47bb-ab9f-c8f088b55cfa/volumes"
Jul 19 17:12:45.688614 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:45.688570    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"]
Jul 19 17:12:45.692799 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:45.692763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:45.693540 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:45.693523    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-352a195c-llfz2"]
Jul 19 17:12:45.839558 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:45.839494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:46.083750 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:46.083648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:46.572101 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:46.572063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:46.632311 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:46.632262    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380a831a-b794-4a85-b78e-9428fbe2b5a8" path="/var/lib/kubelet/pods/380a831a-b794-4a85-b78e-9428fbe2b5a8/volumes"
Jul 19 17:12:46.672084 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:46.672047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:46.771549 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:46.771504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:48.200372 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:48.200336    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:53.904354 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:53.904317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:53.956702 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:53.956666    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:54.036315 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:54.036266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:54.150663 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:54.150630    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:54.253447 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:54.253353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:12:54.615439 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:12:54.615354    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:10.696870 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:10.696842    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:13:10.703720 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:10.703699    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:13:20.600720 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:20.600682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:26.583755 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:26.583709    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:26.684187 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:26.684108    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:26.784009 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:26.783969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:42.689620 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:42.689586    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:48.469960 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:48.469923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:48.531102 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:48.531065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:48.622423 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:48.622391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:48.724490 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:48.724401    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:13:59.202486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:13:59.202451    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:00.055201 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:00.055160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:03.821752 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:03.821678    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:06.027033 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:06.026992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:06.134053 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:06.134022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:06.237574 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:06.237540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:06.327723 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:06.327641    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:12.283023 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:12.282984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:12.381055 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:12.381017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:12.481525 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:12.481487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:14.204228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:14.204191    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:19.843497 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:19.843464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:19.944830 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:19.944789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:20.584812 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:20.584770    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:26.222578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:26.222543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:26.282720 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:26.282665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:26.479015 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:26.478901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:26.790605 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:26.790521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:26.897561 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:26.897521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:30.062701 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.062660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:30.137167 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.137135    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:30.142537 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.142511    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"]
Jul 19 17:14:30.142901 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.142878    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" containerName="maas-api"
Jul 19 17:14:30.142901 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.142900    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" containerName="maas-api"
Jul 19 17:14:30.143112 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.142934    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380a831a-b794-4a85-b78e-9428fbe2b5a8" containerName="revoke-keys"
Jul 19 17:14:30.143112 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.142942    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="380a831a-b794-4a85-b78e-9428fbe2b5a8" containerName="revoke-keys"
Jul 19 17:14:30.143112 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.143037    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="380a831a-b794-4a85-b78e-9428fbe2b5a8" containerName="revoke-keys"
Jul 19 17:14:30.143112 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.143051    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca682f6-7c31-47bb-ab9f-c8f088b55cfa" containerName="maas-api"
Jul 19 17:14:30.146234 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.146214    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.148738 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.148720    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cf60ac3a-serving-cert\""
Jul 19 17:14:30.160056 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.160023    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"]
Jul 19 17:14:30.202819 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.202787    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:30.269988 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.269953    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b39158-61d8-4dc1-9f93-59e2aa4de701-openshift-service-ca\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.270158 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.270019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsrd\" (UniqueName: \"kubernetes.io/projected/45b39158-61d8-4dc1-9f93-59e2aa4de701-kube-api-access-rmsrd\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.270158 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.270045    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b39158-61d8-4dc1-9f93-59e2aa4de701-maas-api-tls\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.370728 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.370629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsrd\" (UniqueName: \"kubernetes.io/projected/45b39158-61d8-4dc1-9f93-59e2aa4de701-kube-api-access-rmsrd\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.370728 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.370672    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b39158-61d8-4dc1-9f93-59e2aa4de701-maas-api-tls\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.370950 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.370741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b39158-61d8-4dc1-9f93-59e2aa4de701-openshift-service-ca\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.371470 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.371449    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b39158-61d8-4dc1-9f93-59e2aa4de701-openshift-service-ca\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.373251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.373234    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b39158-61d8-4dc1-9f93-59e2aa4de701-maas-api-tls\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.385951 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.385922    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsrd\" (UniqueName: \"kubernetes.io/projected/45b39158-61d8-4dc1-9f93-59e2aa4de701-kube-api-access-rmsrd\") pod \"maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.420227 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.420196    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:30.456567 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.456534    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:30.597502 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.597476    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"]
Jul 19 17:14:30.866945 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.866916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:30.967859 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:30.967812    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:31.315534 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:31.315503    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" event={"ID":"45b39158-61d8-4dc1-9f93-59e2aa4de701","Type":"ContainerStarted","Data":"77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78"}
Jul 19 17:14:31.315966 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:31.315562    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" event={"ID":"45b39158-61d8-4dc1-9f93-59e2aa4de701","Type":"ContainerStarted","Data":"5e0840080488a1ff7d8d64875abf23e226fbe215bc68d417f95b1ccc288b79dc"}
Jul 19 17:14:31.315966 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:31.315607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:31.334147 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:31.334092    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" podStartSLOduration=0.94158855 podStartE2EDuration="1.334075373s" totalImagesPullingTime="392.486823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:14:31.33379053 +0000 UTC m=+2181.239041955" watchObservedRunningTime="2026-07-19 17:14:31.334075373 +0000 UTC m=+2181.239326798"
Jul 19 17:14:33.455616 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:33.455578    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:34.620221 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:34.620187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:35.370834 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:35.370801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:35.468129 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:35.468099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:35.867603 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:35.867571    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:37.327532 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:37.327503    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:41.418262 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.418228    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:41.477588 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.477555    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:41.793138 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.793049    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"]
Jul 19 17:14:41.796908 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.796883    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:41.799425 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.799408    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:14:41.803163 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.803141    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"]
Jul 19 17:14:41.981491 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:41.981457    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17069463-26a2-4f3d-bc03-5289d93c5ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8\" (UID: \"17069463-26a2-4f3d-bc03-5289d93c5ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:42.082960 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:42.082843    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17069463-26a2-4f3d-bc03-5289d93c5ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8\" (UID: \"17069463-26a2-4f3d-bc03-5289d93c5ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:42.083701 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:42.083679    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17069463-26a2-4f3d-bc03-5289d93c5ef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8\" (UID: \"17069463-26a2-4f3d-bc03-5289d93c5ef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:42.110905 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:42.110866    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:42.252638 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:42.252609    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"]
Jul 19 17:14:42.256360 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:14:42.256330    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17069463_26a2_4f3d_bc03_5289d93c5ef0.slice/crio-c96aa46ceb63c889614737fc235543bdfe914a5e6ed6ce71f7003c5005f81476 WatchSource:0}: Error finding container c96aa46ceb63c889614737fc235543bdfe914a5e6ed6ce71f7003c5005f81476: Status 404 returned error can't find the container with id c96aa46ceb63c889614737fc235543bdfe914a5e6ed6ce71f7003c5005f81476
Jul 19 17:14:42.375029 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:42.374939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8" event={"ID":"17069463-26a2-4f3d-bc03-5289d93c5ef0","Type":"ContainerStarted","Data":"c96aa46ceb63c889614737fc235543bdfe914a5e6ed6ce71f7003c5005f81476"}
Jul 19 17:14:43.382317 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:43.382200    2584 generic.go:356] "Generic (PLEG): container finished" podID="17069463-26a2-4f3d-bc03-5289d93c5ef0" containerID="68468d747e5e8caf2df92e99a247befb0dc45082c522213efa4c4c0611c920e4" exitCode=0
Jul 19 17:14:43.382317 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:43.382297    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8" event={"ID":"17069463-26a2-4f3d-bc03-5289d93c5ef0","Type":"ContainerDied","Data":"68468d747e5e8caf2df92e99a247befb0dc45082c522213efa4c4c0611c920e4"}
Jul 19 17:14:44.524990 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:44.524961    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:44.706443 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:44.706396    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17069463-26a2-4f3d-bc03-5289d93c5ef0-maas-api-service-ca\") pod \"17069463-26a2-4f3d-bc03-5289d93c5ef0\" (UID: \"17069463-26a2-4f3d-bc03-5289d93c5ef0\") "
Jul 19 17:14:44.706767 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:44.706744    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17069463-26a2-4f3d-bc03-5289d93c5ef0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17069463-26a2-4f3d-bc03-5289d93c5ef0" (UID: "17069463-26a2-4f3d-bc03-5289d93c5ef0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:14:44.807475 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:44.807436    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17069463-26a2-4f3d-bc03-5289d93c5ef0-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:14:45.393470 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:45.393443    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"
Jul 19 17:14:45.393648 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:45.393490    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8" event={"ID":"17069463-26a2-4f3d-bc03-5289d93c5ef0","Type":"ContainerDied","Data":"c96aa46ceb63c889614737fc235543bdfe914a5e6ed6ce71f7003c5005f81476"}
Jul 19 17:14:45.393648 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:45.393536    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96aa46ceb63c889614737fc235543bdfe914a5e6ed6ce71f7003c5005f81476"
Jul 19 17:14:52.153767 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.153730    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"]
Jul 19 17:14:52.154250 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.153951    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" podUID="45b39158-61d8-4dc1-9f93-59e2aa4de701" containerName="maas-api" containerID="cri-o://77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78" gracePeriod=30
Jul 19 17:14:52.175602 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.175572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:52.400587 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.400563    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:52.430012 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.429930    2584 generic.go:356] "Generic (PLEG): container finished" podID="45b39158-61d8-4dc1-9f93-59e2aa4de701" containerID="77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78" exitCode=0
Jul 19 17:14:52.430012 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.429981    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"
Jul 19 17:14:52.430195 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.430014    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" event={"ID":"45b39158-61d8-4dc1-9f93-59e2aa4de701","Type":"ContainerDied","Data":"77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78"}
Jul 19 17:14:52.430195 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.430083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" event={"ID":"45b39158-61d8-4dc1-9f93-59e2aa4de701","Type":"ContainerDied","Data":"5e0840080488a1ff7d8d64875abf23e226fbe215bc68d417f95b1ccc288b79dc"}
Jul 19 17:14:52.430195 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.430111    2584 scope.go:117] "RemoveContainer" containerID="77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78"
Jul 19 17:14:52.439489 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.439468    2584 scope.go:117] "RemoveContainer" containerID="77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78"
Jul 19 17:14:52.439781 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:14:52.439764    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78\": container with ID starting with 77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78 not found: ID does not exist" containerID="77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78"
Jul 19 17:14:52.439875 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.439789    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78"} err="failed to get container status \"77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78\": rpc error: code = NotFound desc = could not find container \"77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78\": container with ID starting with 77b191385083d11fea66c73e1237d431670b4a04515a01fd16cb6844f8b75d78 not found: ID does not exist"
Jul 19 17:14:52.469268 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.469240    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b39158-61d8-4dc1-9f93-59e2aa4de701-maas-api-tls\") pod \"45b39158-61d8-4dc1-9f93-59e2aa4de701\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") "
Jul 19 17:14:52.469430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.469318    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmsrd\" (UniqueName: \"kubernetes.io/projected/45b39158-61d8-4dc1-9f93-59e2aa4de701-kube-api-access-rmsrd\") pod \"45b39158-61d8-4dc1-9f93-59e2aa4de701\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") "
Jul 19 17:14:52.469430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.469401    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b39158-61d8-4dc1-9f93-59e2aa4de701-openshift-service-ca\") pod \"45b39158-61d8-4dc1-9f93-59e2aa4de701\" (UID: \"45b39158-61d8-4dc1-9f93-59e2aa4de701\") "
Jul 19 17:14:52.469754 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.469732    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b39158-61d8-4dc1-9f93-59e2aa4de701-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45b39158-61d8-4dc1-9f93-59e2aa4de701" (UID: "45b39158-61d8-4dc1-9f93-59e2aa4de701"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:14:52.471556 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.471528    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b39158-61d8-4dc1-9f93-59e2aa4de701-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45b39158-61d8-4dc1-9f93-59e2aa4de701" (UID: "45b39158-61d8-4dc1-9f93-59e2aa4de701"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:14:52.471659 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.471566    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b39158-61d8-4dc1-9f93-59e2aa4de701-kube-api-access-rmsrd" (OuterVolumeSpecName: "kube-api-access-rmsrd") pod "45b39158-61d8-4dc1-9f93-59e2aa4de701" (UID: "45b39158-61d8-4dc1-9f93-59e2aa4de701"). InnerVolumeSpecName "kube-api-access-rmsrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:14:52.570204 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.570169    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b39158-61d8-4dc1-9f93-59e2aa4de701-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:14:52.570204 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.570196    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b39158-61d8-4dc1-9f93-59e2aa4de701-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:14:52.570204 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.570205    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmsrd\" (UniqueName: \"kubernetes.io/projected/45b39158-61d8-4dc1-9f93-59e2aa4de701-kube-api-access-rmsrd\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:14:52.745295 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.745199    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"]
Jul 19 17:14:52.746991 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:52.746969    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz"]
Jul 19 17:14:54.633181 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:54.633150    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b39158-61d8-4dc1-9f93-59e2aa4de701" path="/var/lib/kubelet/pods/45b39158-61d8-4dc1-9f93-59e2aa4de701/volumes"
Jul 19 17:14:55.322901 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:55.322844    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cf60ac3a-cdfb86f4f-krkmz" podUID="45b39158-61d8-4dc1-9f93-59e2aa4de701" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": context deadline exceeded"
Jul 19 17:14:56.861486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:56.861449    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"]
Jul 19 17:14:56.863664 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:56.863633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:56.866102 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:56.866078    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cf60ac3a-68fr8"]
Jul 19 17:14:57.246804 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:57.246771    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:14:58.632186 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:14:58.632150    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17069463-26a2-4f3d-bc03-5289d93c5ef0" path="/var/lib/kubelet/pods/17069463-26a2-4f3d-bc03-5289d93c5ef0/volumes"
Jul 19 17:15:00.138553 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138519    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"]
Jul 19 17:15:00.138968 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138880    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b39158-61d8-4dc1-9f93-59e2aa4de701" containerName="maas-api"
Jul 19 17:15:00.138968 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138891    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b39158-61d8-4dc1-9f93-59e2aa4de701" containerName="maas-api"
Jul 19 17:15:00.138968 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138913    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17069463-26a2-4f3d-bc03-5289d93c5ef0" containerName="revoke-keys"
Jul 19 17:15:00.138968 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138920    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="17069463-26a2-4f3d-bc03-5289d93c5ef0" containerName="revoke-keys"
Jul 19 17:15:00.138968 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138972    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b39158-61d8-4dc1-9f93-59e2aa4de701" containerName="maas-api"
Jul 19 17:15:00.139154 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.138980    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="17069463-26a2-4f3d-bc03-5289d93c5ef0" containerName="revoke-keys"
Jul 19 17:15:00.143185 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.143165    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"
Jul 19 17:15:00.146170 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.146142    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:15:00.148328 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.148302    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"]
Jul 19 17:15:00.155487 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.155469    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"
Jul 19 17:15:00.295832 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.295652    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"]
Jul 19 17:15:00.298239 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:15:00.298208    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24f482e_8a01_4b25_8293_f7d12a74a1e8.slice/crio-e1dad700fa0accdb893cb73ac9f42a2cfc6e39af633891143670734064fb23af WatchSource:0}: Error finding container e1dad700fa0accdb893cb73ac9f42a2cfc6e39af633891143670734064fb23af: Status 404 returned error can't find the container with id e1dad700fa0accdb893cb73ac9f42a2cfc6e39af633891143670734064fb23af
Jul 19 17:15:00.483706 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:00.483662    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk" event={"ID":"c24f482e-8a01-4b25-8293-f7d12a74a1e8","Type":"ContainerStarted","Data":"e1dad700fa0accdb893cb73ac9f42a2cfc6e39af633891143670734064fb23af"}
Jul 19 17:15:01.490201 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:01.490159    2584 generic.go:356] "Generic (PLEG): container finished" podID="c24f482e-8a01-4b25-8293-f7d12a74a1e8" containerID="ce034afe9b8bd80e1893761d9b72bc0dbfb03e76de36b4e925ca551388bc066a" exitCode=0
Jul 19 17:15:01.490671 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:01.490255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk" event={"ID":"c24f482e-8a01-4b25-8293-f7d12a74a1e8","Type":"ContainerDied","Data":"ce034afe9b8bd80e1893761d9b72bc0dbfb03e76de36b4e925ca551388bc066a"}
Jul 19 17:15:02.632384 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:02.632355    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"
Jul 19 17:15:03.060082 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.060046    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:03.161041 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.161000    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:03.218161 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.218131    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:03.435476 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.435435    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:03.502081 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.502038    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk" event={"ID":"c24f482e-8a01-4b25-8293-f7d12a74a1e8","Type":"ContainerDied","Data":"e1dad700fa0accdb893cb73ac9f42a2cfc6e39af633891143670734064fb23af"}
Jul 19 17:15:03.502081 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.502073    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741355-gztfk"
Jul 19 17:15:03.502306 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.502092    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1dad700fa0accdb893cb73ac9f42a2cfc6e39af633891143670734064fb23af"
Jul 19 17:15:03.672145 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.672107    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:03.803435 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.803332    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:03.893296 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:03.893251    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:04.026473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:04.026440    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:09.708489 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:09.708454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:09.821022 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:09.820990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:09.936865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:09.936834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:10.069063 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:10.068964    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:15.878508 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:15.878474    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:15.936354 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:15.936319    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:22.712895 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:22.712862    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:23.031997 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:23.031907    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:23.147671 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:23.147615    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:26.275115 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.275082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:26.357805 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.357744    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"]
Jul 19 17:15:26.358342 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.358314    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24f482e-8a01-4b25-8293-f7d12a74a1e8" containerName="cleanup"
Jul 19 17:15:26.358342 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.358340    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24f482e-8a01-4b25-8293-f7d12a74a1e8" containerName="cleanup"
Jul 19 17:15:26.358514 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.358429    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24f482e-8a01-4b25-8293-f7d12a74a1e8" containerName="cleanup"
Jul 19 17:15:26.359667 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.359633    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:26.361501 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.361480    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.364053 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.364031    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-68c77091-serving-cert\""
Jul 19 17:15:26.371787 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.371758    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"]
Jul 19 17:15:26.421600 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.421564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:26.464994 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.464961    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.465156 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.465013    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vwrf\" (UniqueName: \"kubernetes.io/projected/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-kube-api-access-7vwrf\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.465156 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.465042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-openshift-service-ca\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.566344 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.566235    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.566344 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.566319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vwrf\" (UniqueName: \"kubernetes.io/projected/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-kube-api-access-7vwrf\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.566532 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.566361    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-openshift-service-ca\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.566532 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:15:26.566387    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-serving-cert: secret "maas-api-e2e-mt-68c77091-serving-cert" not found
Jul 19 17:15:26.566532 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:15:26.566451    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls podName:2435e8e9-32ff-4cc6-9756-0ea61f6864ed nodeName:}" failed. No retries permitted until 2026-07-19 17:15:27.066434292 +0000 UTC m=+2236.971685694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls") pod "maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" (UID: "2435e8e9-32ff-4cc6-9756-0ea61f6864ed") : secret "maas-api-e2e-mt-68c77091-serving-cert" not found
Jul 19 17:15:26.566955 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.566937    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-openshift-service-ca\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.575145 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.575100    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vwrf\" (UniqueName: \"kubernetes.io/projected/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-kube-api-access-7vwrf\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:26.620701 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:26.620657    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:27.023708 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.023675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:27.070392 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.070343    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:27.073093 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.073060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls\") pod \"maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:27.122960 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.122924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:27.273700 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.273668    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:27.417971 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.417942    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"]
Jul 19 17:15:27.420582 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:15:27.420556    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2435e8e9_32ff_4cc6_9756_0ea61f6864ed.slice/crio-d389bf143da42eb6a2c183e7c6148d335f13c7b06eb714274fbca7b94c22467b WatchSource:0}: Error finding container d389bf143da42eb6a2c183e7c6148d335f13c7b06eb714274fbca7b94c22467b: Status 404 returned error can't find the container with id d389bf143da42eb6a2c183e7c6148d335f13c7b06eb714274fbca7b94c22467b
Jul 19 17:15:27.421873 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.421850    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 17:15:27.621747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:27.621657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" event={"ID":"2435e8e9-32ff-4cc6-9756-0ea61f6864ed","Type":"ContainerStarted","Data":"d389bf143da42eb6a2c183e7c6148d335f13c7b06eb714274fbca7b94c22467b"}
Jul 19 17:15:28.633380 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:28.633351    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:28.633803 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:28.633397    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" event={"ID":"2435e8e9-32ff-4cc6-9756-0ea61f6864ed","Type":"ContainerStarted","Data":"31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b"}
Jul 19 17:15:28.645976 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:28.645933    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" podStartSLOduration=2.274879146 podStartE2EDuration="2.645917941s" totalImagesPullingTime="371.038795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:15:28.643576261 +0000 UTC m=+2238.548827686" watchObservedRunningTime="2026-07-19 17:15:28.645917941 +0000 UTC m=+2238.551169364"
Jul 19 17:15:30.797177 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:30.797143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:34.640425 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:34.640344    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:15:41.532430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.532387    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"]
Jul 19 17:15:41.542383 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.542350    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.546060 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.546026    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"]
Jul 19 17:15:41.546255 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.546087    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-68c77091\"/\"e2e-lifecycle-model-68c77091-kserve-self-signed-certs\""
Jul 19 17:15:41.546383 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.546181    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-68c77091\"/\"kube-root-ca.crt\""
Jul 19 17:15:41.546667 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.546649    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-68c77091\"/\"openshift-service-ca.crt\""
Jul 19 17:15:41.546773 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.546754    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-68c77091\"/\"default-dockercfg-9qfgg\""
Jul 19 17:15:41.574873 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.574844    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:15:41.704886 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.704849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-home\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.704886 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.704891    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-tmp-dir\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.705116 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.704924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-model-cache\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.705116 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.704998    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-dshm\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.705116 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.705025    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8mc\" (UniqueName: \"kubernetes.io/projected/70f35b49-0000-406a-aab9-5357183b15a2-kube-api-access-bl8mc\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.705116 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.705063    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f35b49-0000-406a-aab9-5357183b15a2-tls-certs\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.805984    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-home\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-tmp-dir\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806045    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-model-cache\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806072    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-dshm\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806088    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8mc\" (UniqueName: \"kubernetes.io/projected/70f35b49-0000-406a-aab9-5357183b15a2-kube-api-access-bl8mc\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806602 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f35b49-0000-406a-aab9-5357183b15a2-tls-certs\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806602 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806487    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-model-cache\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806602 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-home\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.806739 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.806600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-tmp-dir\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.808622 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.808602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-dshm\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.808906 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.808886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f35b49-0000-406a-aab9-5357183b15a2-tls-certs\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.813569 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.813546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8mc\" (UniqueName: \"kubernetes.io/projected/70f35b49-0000-406a-aab9-5357183b15a2-kube-api-access-bl8mc\") pod \"e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") " pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.855204 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.855164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:41.996834 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:41.996808    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"]
Jul 19 17:15:41.999467 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:15:41.999437    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70f35b49_0000_406a_aab9_5357183b15a2.slice/crio-b923d4a059bce9b35b8b83df43756447d712ca2d350e4bb01e84c961cc636ba7 WatchSource:0}: Error finding container b923d4a059bce9b35b8b83df43756447d712ca2d350e4bb01e84c961cc636ba7: Status 404 returned error can't find the container with id b923d4a059bce9b35b8b83df43756447d712ca2d350e4bb01e84c961cc636ba7
Jul 19 17:15:42.702134 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:42.702098    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w" event={"ID":"70f35b49-0000-406a-aab9-5357183b15a2","Type":"ContainerStarted","Data":"618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b"}
Jul 19 17:15:42.702134 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:42.702139    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w" event={"ID":"70f35b49-0000-406a-aab9-5357183b15a2","Type":"ContainerStarted","Data":"b923d4a059bce9b35b8b83df43756447d712ca2d350e4bb01e84c961cc636ba7"}
Jul 19 17:15:42.718081 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:42.718031    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w" podStartSLOduration=1.718013557 podStartE2EDuration="1.718013557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:15:42.716864377 +0000 UTC m=+2252.622115802" watchObservedRunningTime="2026-07-19 17:15:42.718013557 +0000 UTC m=+2252.623264997"
Jul 19 17:15:51.855900 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:51.855865    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:51.856381 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:51.855914    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:51.868710 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:51.868683    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:15:52.763880 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:15:52.763852    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:16:09.263290 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.263251    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"]
Jul 19 17:16:09.266968 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.266949    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:09.269530 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.269507    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:16:09.273450 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.273427    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"]
Jul 19 17:16:09.360181 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.360133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809471dc-daad-47e8-953a-8712ef65b594-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-68c77091-mg5gn\" (UID: \"809471dc-daad-47e8-953a-8712ef65b594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:09.461267 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.461224    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809471dc-daad-47e8-953a-8712ef65b594-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-68c77091-mg5gn\" (UID: \"809471dc-daad-47e8-953a-8712ef65b594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:09.461889 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.461865    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809471dc-daad-47e8-953a-8712ef65b594-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-68c77091-mg5gn\" (UID: \"809471dc-daad-47e8-953a-8712ef65b594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:09.580225 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.580132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:09.726582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.726559    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"]
Jul 19 17:16:09.728742 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:16:09.728526    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809471dc_daad_47e8_953a_8712ef65b594.slice/crio-35f647e418da34c6e04aa0ca581c8b41f41fa888ed974341f37d04095cc42e55 WatchSource:0}: Error finding container 35f647e418da34c6e04aa0ca581c8b41f41fa888ed974341f37d04095cc42e55: Status 404 returned error can't find the container with id 35f647e418da34c6e04aa0ca581c8b41f41fa888ed974341f37d04095cc42e55
Jul 19 17:16:09.843357 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:09.843239    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn" event={"ID":"809471dc-daad-47e8-953a-8712ef65b594","Type":"ContainerStarted","Data":"35f647e418da34c6e04aa0ca581c8b41f41fa888ed974341f37d04095cc42e55"}
Jul 19 17:16:10.848915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:10.848826    2584 generic.go:356] "Generic (PLEG): container finished" podID="809471dc-daad-47e8-953a-8712ef65b594" containerID="c86a59230800c1e4005479405a3c5505a5309e05658dd9b515a7ddc7f9cf0b04" exitCode=0
Jul 19 17:16:10.848915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:10.848906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn" event={"ID":"809471dc-daad-47e8-953a-8712ef65b594","Type":"ContainerDied","Data":"c86a59230800c1e4005479405a3c5505a5309e05658dd9b515a7ddc7f9cf0b04"}
Jul 19 17:16:11.992458 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:11.992431    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:12.083441 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:12.083409    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809471dc-daad-47e8-953a-8712ef65b594-maas-api-service-ca\") pod \"809471dc-daad-47e8-953a-8712ef65b594\" (UID: \"809471dc-daad-47e8-953a-8712ef65b594\") "
Jul 19 17:16:12.083870 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:12.083844    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809471dc-daad-47e8-953a-8712ef65b594-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "809471dc-daad-47e8-953a-8712ef65b594" (UID: "809471dc-daad-47e8-953a-8712ef65b594"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:16:12.184897 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:12.184859    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809471dc-daad-47e8-953a-8712ef65b594-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:12.860368 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:12.860330    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"
Jul 19 17:16:12.860617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:12.860375    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn" event={"ID":"809471dc-daad-47e8-953a-8712ef65b594","Type":"ContainerDied","Data":"35f647e418da34c6e04aa0ca581c8b41f41fa888ed974341f37d04095cc42e55"}
Jul 19 17:16:12.860617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:12.860423    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f647e418da34c6e04aa0ca581c8b41f41fa888ed974341f37d04095cc42e55"
Jul 19 17:16:24.659393 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.659355    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"]
Jul 19 17:16:24.659859 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.659570    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" podUID="2435e8e9-32ff-4cc6-9756-0ea61f6864ed" containerName="maas-api" containerID="cri-o://31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b" gracePeriod=30
Jul 19 17:16:24.686108 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.686068    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:16:24.918563 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.918499    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:16:24.927342 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.927307    2584 generic.go:356] "Generic (PLEG): container finished" podID="2435e8e9-32ff-4cc6-9756-0ea61f6864ed" containerID="31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b" exitCode=0
Jul 19 17:16:24.927529 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.927328    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" event={"ID":"2435e8e9-32ff-4cc6-9756-0ea61f6864ed","Type":"ContainerDied","Data":"31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b"}
Jul 19 17:16:24.927529 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.927360    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"
Jul 19 17:16:24.927529 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.927409    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl" event={"ID":"2435e8e9-32ff-4cc6-9756-0ea61f6864ed","Type":"ContainerDied","Data":"d389bf143da42eb6a2c183e7c6148d335f13c7b06eb714274fbca7b94c22467b"}
Jul 19 17:16:24.927529 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.927441    2584 scope.go:117] "RemoveContainer" containerID="31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b"
Jul 19 17:16:24.937348 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.937319    2584 scope.go:117] "RemoveContainer" containerID="31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b"
Jul 19 17:16:24.937690 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:16:24.937660    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b\": container with ID starting with 31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b not found: ID does not exist" containerID="31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b"
Jul 19 17:16:24.937765 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:24.937703    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b"} err="failed to get container status \"31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b\": rpc error: code = NotFound desc = could not find container \"31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b\": container with ID starting with 31944d44018fe9fb06f74509823ddcacf8608a380650b93734bcdf656e53345b not found: ID does not exist"
Jul 19 17:16:25.106578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.106538    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls\") pod \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") "
Jul 19 17:16:25.106759 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.106631    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-openshift-service-ca\") pod \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") "
Jul 19 17:16:25.106759 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.106667    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vwrf\" (UniqueName: \"kubernetes.io/projected/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-kube-api-access-7vwrf\") pod \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\" (UID: \"2435e8e9-32ff-4cc6-9756-0ea61f6864ed\") "
Jul 19 17:16:25.107082 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.107047    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2435e8e9-32ff-4cc6-9756-0ea61f6864ed" (UID: "2435e8e9-32ff-4cc6-9756-0ea61f6864ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:16:25.108990 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.108962    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-kube-api-access-7vwrf" (OuterVolumeSpecName: "kube-api-access-7vwrf") pod "2435e8e9-32ff-4cc6-9756-0ea61f6864ed" (UID: "2435e8e9-32ff-4cc6-9756-0ea61f6864ed"). InnerVolumeSpecName "kube-api-access-7vwrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:16:25.108990 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.108981    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2435e8e9-32ff-4cc6-9756-0ea61f6864ed" (UID: "2435e8e9-32ff-4cc6-9756-0ea61f6864ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:16:25.207854 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.207764    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:25.207854 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.207797    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vwrf\" (UniqueName: \"kubernetes.io/projected/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-kube-api-access-7vwrf\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:25.207854 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.207807    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2435e8e9-32ff-4cc6-9756-0ea61f6864ed-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:25.248027 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.247996    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"]
Jul 19 17:16:25.250204 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:25.250182    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-68c77091-78cb795cc5-gh9rl"]
Jul 19 17:16:26.632413 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:26.632379    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2435e8e9-32ff-4cc6-9756-0ea61f6864ed" path="/var/lib/kubelet/pods/2435e8e9-32ff-4cc6-9756-0ea61f6864ed/volumes"
Jul 19 17:16:29.331745 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:29.331704    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"]
Jul 19 17:16:29.334219 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:29.334186    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-68c77091-mg5gn"]
Jul 19 17:16:29.334749 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:29.334723    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:16:30.632697 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:30.632660    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="809471dc-daad-47e8-953a-8712ef65b594" path="/var/lib/kubelet/pods/809471dc-daad-47e8-953a-8712ef65b594/volumes"
Jul 19 17:16:31.584831 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:31.584794    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:16:36.568876 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:36.568837    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"]
Jul 19 17:16:36.610293 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:36.610259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:16:51.589823 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.589771    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w" podUID="70f35b49-0000-406a-aab9-5357183b15a2" containerName="main" containerID="cri-o://618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b" gracePeriod=45
Jul 19 17:16:51.851062 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.850997    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:16:51.954377 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954341    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl8mc\" (UniqueName: \"kubernetes.io/projected/70f35b49-0000-406a-aab9-5357183b15a2-kube-api-access-bl8mc\") pod \"70f35b49-0000-406a-aab9-5357183b15a2\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") "
Jul 19 17:16:51.954573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954392    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-dshm\") pod \"70f35b49-0000-406a-aab9-5357183b15a2\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") "
Jul 19 17:16:51.954573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954424    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f35b49-0000-406a-aab9-5357183b15a2-tls-certs\") pod \"70f35b49-0000-406a-aab9-5357183b15a2\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") "
Jul 19 17:16:51.954573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954449    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-model-cache\") pod \"70f35b49-0000-406a-aab9-5357183b15a2\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") "
Jul 19 17:16:51.954573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954501    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-tmp-dir\") pod \"70f35b49-0000-406a-aab9-5357183b15a2\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") "
Jul 19 17:16:51.954793 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954595    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-home\") pod \"70f35b49-0000-406a-aab9-5357183b15a2\" (UID: \"70f35b49-0000-406a-aab9-5357183b15a2\") "
Jul 19 17:16:51.954793 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954758    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70f35b49-0000-406a-aab9-5357183b15a2" (UID: "70f35b49-0000-406a-aab9-5357183b15a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:16:51.954793 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954780    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "70f35b49-0000-406a-aab9-5357183b15a2" (UID: "70f35b49-0000-406a-aab9-5357183b15a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:16:51.954946 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954870    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:51.954946 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954884    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:51.954946 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.954889    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-home" (OuterVolumeSpecName: "home") pod "70f35b49-0000-406a-aab9-5357183b15a2" (UID: "70f35b49-0000-406a-aab9-5357183b15a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:16:51.956879 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.956855    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f35b49-0000-406a-aab9-5357183b15a2-kube-api-access-bl8mc" (OuterVolumeSpecName: "kube-api-access-bl8mc") pod "70f35b49-0000-406a-aab9-5357183b15a2" (UID: "70f35b49-0000-406a-aab9-5357183b15a2"). InnerVolumeSpecName "kube-api-access-bl8mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:16:51.957123 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.957107    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-dshm" (OuterVolumeSpecName: "dshm") pod "70f35b49-0000-406a-aab9-5357183b15a2" (UID: "70f35b49-0000-406a-aab9-5357183b15a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:16:51.957181 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:51.957121    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f35b49-0000-406a-aab9-5357183b15a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70f35b49-0000-406a-aab9-5357183b15a2" (UID: "70f35b49-0000-406a-aab9-5357183b15a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:16:52.055647 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.055601    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f35b49-0000-406a-aab9-5357183b15a2-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:52.055647 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.055635    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:52.055647 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.055644    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl8mc\" (UniqueName: \"kubernetes.io/projected/70f35b49-0000-406a-aab9-5357183b15a2-kube-api-access-bl8mc\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:52.055903 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.055666    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f35b49-0000-406a-aab9-5357183b15a2-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:16:52.066933 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.066896    2584 generic.go:356] "Generic (PLEG): container finished" podID="70f35b49-0000-406a-aab9-5357183b15a2" containerID="618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b" exitCode=0
Jul 19 17:16:52.067091 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.066947    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"
Jul 19 17:16:52.067091 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.066989    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w" event={"ID":"70f35b49-0000-406a-aab9-5357183b15a2","Type":"ContainerDied","Data":"618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b"}
Jul 19 17:16:52.067091 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.067035    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w" event={"ID":"70f35b49-0000-406a-aab9-5357183b15a2","Type":"ContainerDied","Data":"b923d4a059bce9b35b8b83df43756447d712ca2d350e4bb01e84c961cc636ba7"}
Jul 19 17:16:52.067091 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.067057    2584 scope.go:117] "RemoveContainer" containerID="618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b"
Jul 19 17:16:52.076765 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.076748    2584 scope.go:117] "RemoveContainer" containerID="618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b"
Jul 19 17:16:52.077049 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:16:52.077029    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b\": container with ID starting with 618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b not found: ID does not exist" containerID="618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b"
Jul 19 17:16:52.077088 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.077059    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b"} err="failed to get container status \"618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b\": rpc error: code = NotFound desc = could not find container \"618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b\": container with ID starting with 618185f4a2e3911e00b8d16561804b7bb5ebe5a21a32ae47c7b1f0ffa9e59b8b not found: ID does not exist"
Jul 19 17:16:52.088086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.088057    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"]
Jul 19 17:16:52.091697 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.091671    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-68c77091/e2e-lifecycle-model-68c77091-kserve-bd5b6dcc9-t925w"]
Jul 19 17:16:52.632503 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:16:52.632474    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f35b49-0000-406a-aab9-5357183b15a2" path="/var/lib/kubelet/pods/70f35b49-0000-406a-aab9-5357183b15a2/volumes"
Jul 19 17:17:10.845019 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:10.844942    2584 scope.go:117] "RemoveContainer" containerID="37a4ca1c71251375f21e2e1a6d53f463ddffdffb014d333bc15bf71757cef7df"
Jul 19 17:17:10.853486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:10.853464    2584 scope.go:117] "RemoveContainer" containerID="e553a9d122dcc8e92cc7599945fb1c58972605dac44cd27fd8ff5499cd5b6807"
Jul 19 17:17:13.275830 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:13.275793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:13.598947 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:13.598861    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:19.042245 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:19.042202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:19.101669 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:19.101619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:19.325121 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:19.325028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:19.580128 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:19.580036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:20.827712 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:20.827659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:20.926484 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:20.926451    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:21.030251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:21.030187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:21.864654 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:21.864613    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:21.964625 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:21.964574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:22.064688 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:22.064623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:23.544252 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:23.544208    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:26.619596 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:26.619531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:26.717305 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:26.717250    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:29.116430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:29.116380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:29.168543 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:29.168510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:29.817859 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:29.817823    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:35.423637 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:35.423603    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:35.530928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:35.530892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:35.624033 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:35.623995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:46.195672 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:46.195632    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:17:46.548210 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:17:46.548098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:07.579432 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:07.579391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:10.725403 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:10.725374    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:18:10.733017 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:10.732993    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:18:10.887464 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:10.887426    2584 scope.go:117] "RemoveContainer" containerID="8cbd85d1c6db9f975268ae020752bf492fc1b1302ba0ad26e88b31c3ea9fc01b"
Jul 19 17:18:13.938568 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:13.938540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:14.036995 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:14.036956    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:14.308489 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:14.308394    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:14.767637 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:14.767599    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:14.865842 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:14.865808    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:18.158470 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.158425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:18.251053 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.251018    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:18.252124 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252102    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"]
Jul 19 17:18:18.252540 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252526    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f35b49-0000-406a-aab9-5357183b15a2" containerName="main"
Jul 19 17:18:18.252583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252542    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f35b49-0000-406a-aab9-5357183b15a2" containerName="main"
Jul 19 17:18:18.252583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252562    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2435e8e9-32ff-4cc6-9756-0ea61f6864ed" containerName="maas-api"
Jul 19 17:18:18.252583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252569    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2435e8e9-32ff-4cc6-9756-0ea61f6864ed" containerName="maas-api"
Jul 19 17:18:18.252679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252588    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="809471dc-daad-47e8-953a-8712ef65b594" containerName="revoke-keys"
Jul 19 17:18:18.252679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252594    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="809471dc-daad-47e8-953a-8712ef65b594" containerName="revoke-keys"
Jul 19 17:18:18.252679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252654    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2435e8e9-32ff-4cc6-9756-0ea61f6864ed" containerName="maas-api"
Jul 19 17:18:18.252679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252663    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="809471dc-daad-47e8-953a-8712ef65b594" containerName="revoke-keys"
Jul 19 17:18:18.252679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.252669    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f35b49-0000-406a-aab9-5357183b15a2" containerName="main"
Jul 19 17:18:18.255618 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.255602    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.258347 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.258313    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7dceef-serving-cert\""
Jul 19 17:18:18.264835 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.264813    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"]
Jul 19 17:18:18.315923 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.315886    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d98552d-ac48-4da0-a801-6466f816a5bf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.316100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.315949    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2dn\" (UniqueName: \"kubernetes.io/projected/6d98552d-ac48-4da0-a801-6466f816a5bf-kube-api-access-ch2dn\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.316147 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.316101    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d98552d-ac48-4da0-a801-6466f816a5bf-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.335519 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.335486    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:18.416837 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.416750    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d98552d-ac48-4da0-a801-6466f816a5bf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.416837 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.416794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2dn\" (UniqueName: \"kubernetes.io/projected/6d98552d-ac48-4da0-a801-6466f816a5bf-kube-api-access-ch2dn\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.417069 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.416857    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d98552d-ac48-4da0-a801-6466f816a5bf-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.417495 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.417472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d98552d-ac48-4da0-a801-6466f816a5bf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.419510 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.419489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d98552d-ac48-4da0-a801-6466f816a5bf-maas-api-tls\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.424914 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.424889    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2dn\" (UniqueName: \"kubernetes.io/projected/6d98552d-ac48-4da0-a801-6466f816a5bf-kube-api-access-ch2dn\") pod \"maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.437807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.437785    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:18.566887 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.566858    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:18.716563 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.716539    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"]
Jul 19 17:18:18.718710 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:18:18.718669    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d98552d_ac48_4da0_a801_6466f816a5bf.slice/crio-657089a9c8cbce31730b5b238badeba9aee6f73afde3243bd454ea8a02bb7241 WatchSource:0}: Error finding container 657089a9c8cbce31730b5b238badeba9aee6f73afde3243bd454ea8a02bb7241: Status 404 returned error can't find the container with id 657089a9c8cbce31730b5b238badeba9aee6f73afde3243bd454ea8a02bb7241
Jul 19 17:18:18.829264 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.829230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:18.929291 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:18.929246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:19.514310 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:19.514200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8" event={"ID":"6d98552d-ac48-4da0-a801-6466f816a5bf","Type":"ContainerStarted","Data":"71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5"}
Jul 19 17:18:19.514693 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:19.514323    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8" event={"ID":"6d98552d-ac48-4da0-a801-6466f816a5bf","Type":"ContainerStarted","Data":"657089a9c8cbce31730b5b238badeba9aee6f73afde3243bd454ea8a02bb7241"}
Jul 19 17:18:19.514693 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:19.514353    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:19.531002 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:19.530943    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8" podStartSLOduration=1.047273232 podStartE2EDuration="1.530926006s" totalImagesPullingTime="483.652774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:18:19.529080927 +0000 UTC m=+2409.434332350" watchObservedRunningTime="2026-07-19 17:18:19.530926006 +0000 UTC m=+2409.436177429"
Jul 19 17:18:22.803352 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:22.803314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:23.207399 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:23.207369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:23.524880 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:23.524790    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:24.042867 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:24.042832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:25.526208 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:25.526171    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:18:26.307465 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.307427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:26.407504 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.407472    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:26.533216 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.533185    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"]
Jul 19 17:18:26.537196 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.537173    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.539611 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.539586    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ec4179-serving-cert\""
Jul 19 17:18:26.549006 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.548979    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"]
Jul 19 17:18:26.567596 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.567510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:26.587667 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.587634    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.587861 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.587714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.587861 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.587742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snmwd\" (UniqueName: \"kubernetes.io/projected/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-kube-api-access-snmwd\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.688458 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.688420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.688674 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.688511    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.688674 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.688562    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snmwd\" (UniqueName: \"kubernetes.io/projected/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-kube-api-access-snmwd\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.688969 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:18:26.688678    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-serving-cert: secret "maas-api-e2e-inf-b-ec4179-serving-cert" not found
Jul 19 17:18:26.689054 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:18:26.689033    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls podName:c5f0c8b1-e8ec-4e96-8c5c-29971c581963 nodeName:}" failed. No retries permitted until 2026-07-19 17:18:27.189010428 +0000 UTC m=+2417.094261834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls") pod "maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" (UID: "c5f0c8b1-e8ec-4e96-8c5c-29971c581963") : secret "maas-api-e2e-inf-b-ec4179-serving-cert" not found
Jul 19 17:18:26.689175 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.689154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.699774 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.699744    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snmwd\" (UniqueName: \"kubernetes.io/projected/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-kube-api-access-snmwd\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:26.859331 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:26.859224    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:27.183745 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:27.183690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:27.193504 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:27.193475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:27.196098 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:27.196071    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls\") pod \"maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:27.286396 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:27.286362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:27.451179 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:27.451078    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:27.605783 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:27.605753    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"]
Jul 19 17:18:27.608052 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:18:27.608021    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f0c8b1_e8ec_4e96_8c5c_29971c581963.slice/crio-a190e53f7b1065d80412ff1ec187208cd391c5c7953e377d9e753d4634a688a4 WatchSource:0}: Error finding container a190e53f7b1065d80412ff1ec187208cd391c5c7953e377d9e753d4634a688a4: Status 404 returned error can't find the container with id a190e53f7b1065d80412ff1ec187208cd391c5c7953e377d9e753d4634a688a4
Jul 19 17:18:28.565665 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:28.565619    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" event={"ID":"c5f0c8b1-e8ec-4e96-8c5c-29971c581963","Type":"ContainerStarted","Data":"01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900"}
Jul 19 17:18:28.565871 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:28.565689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" event={"ID":"c5f0c8b1-e8ec-4e96-8c5c-29971c581963","Type":"ContainerStarted","Data":"a190e53f7b1065d80412ff1ec187208cd391c5c7953e377d9e753d4634a688a4"}
Jul 19 17:18:28.565871 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:28.565831    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:28.581182 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:28.581116    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" podStartSLOduration=2.196076853 podStartE2EDuration="2.581094036s" totalImagesPullingTime="385.017183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:18:28.579635002 +0000 UTC m=+2418.484886428" watchObservedRunningTime="2026-07-19 17:18:28.581094036 +0000 UTC m=+2418.486345465"
Jul 19 17:18:30.820418 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:30.820388    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:34.576463 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:34.576388    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:18:39.587472 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:39.587437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:39.688554 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:39.688515    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:48.745100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:48.745063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:48.844954 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:48.844917    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:53.041354 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.041315    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"]
Jul 19 17:18:53.045031 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.045009    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.048637 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.048609    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7dceef\"/\"kube-root-ca.crt\""
Jul 19 17:18:53.048767 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.048692    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7dceef\"/\"test-model-7dceef-kserve-self-signed-certs\""
Jul 19 17:18:53.048767 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.048720    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7dceef\"/\"default-dockercfg-g4z6g\""
Jul 19 17:18:53.048767 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.048724    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7dceef\"/\"openshift-service-ca.crt\""
Jul 19 17:18:53.054522 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.054497    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"]
Jul 19 17:18:53.123174 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.123142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:18:53.130582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.130551    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-model-cache\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.130731 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.130605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l842\" (UniqueName: \"kubernetes.io/projected/a6112f29-0888-4b85-b187-f9b3f8718162-kube-api-access-9l842\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.130731 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.130657    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-home\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.130731 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.130689    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-dshm\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.130731 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.130716    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6112f29-0888-4b85-b187-f9b3f8718162-tls-certs\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.130864 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.130733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-tmp-dir\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.231549 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231513    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-model-cache\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.231747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l842\" (UniqueName: \"kubernetes.io/projected/a6112f29-0888-4b85-b187-f9b3f8718162-kube-api-access-9l842\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.231747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231617    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-home\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.231747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231638    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-dshm\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.231747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231674    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6112f29-0888-4b85-b187-f9b3f8718162-tls-certs\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.231747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-tmp-dir\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.232006 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231983    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-model-cache\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.232006 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.231996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-home\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.232092 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.232079    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-tmp-dir\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.234217 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.234193    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-dshm\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.234454 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.234437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6112f29-0888-4b85-b187-f9b3f8718162-tls-certs\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.239466 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.239442    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l842\" (UniqueName: \"kubernetes.io/projected/a6112f29-0888-4b85-b187-f9b3f8718162-kube-api-access-9l842\") pod \"test-model-7dceef-kserve-57549bbb65-h7hrl\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") " pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.355670 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.355581    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:18:53.495249 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.495202    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"]
Jul 19 17:18:53.497639 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:18:53.497609    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6112f29_0888_4b85_b187_f9b3f8718162.slice/crio-8f1611b83672a8b87cac6120f80db78808dfb2d83e2ef204006f1bbf78efc460 WatchSource:0}: Error finding container 8f1611b83672a8b87cac6120f80db78808dfb2d83e2ef204006f1bbf78efc460: Status 404 returned error can't find the container with id 8f1611b83672a8b87cac6120f80db78808dfb2d83e2ef204006f1bbf78efc460
Jul 19 17:18:53.694710 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.694662    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl" event={"ID":"a6112f29-0888-4b85-b187-f9b3f8718162","Type":"ContainerStarted","Data":"105bd6ff7ffcef106a9a61fe42866d96e95980e8ff4f76fbee5f82d77567cd70"}
Jul 19 17:18:53.694710 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.694716    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl" event={"ID":"a6112f29-0888-4b85-b187-f9b3f8718162","Type":"ContainerStarted","Data":"8f1611b83672a8b87cac6120f80db78808dfb2d83e2ef204006f1bbf78efc460"}
Jul 19 17:18:53.715307 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:18:53.714633    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl" podStartSLOduration=0.714611449 podStartE2EDuration="714.611449ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:18:53.71076415 +0000 UTC m=+2443.616015586" watchObservedRunningTime="2026-07-19 17:18:53.714611449 +0000 UTC m=+2443.619862873"
Jul 19 17:19:03.355972 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:03.355914    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:19:03.355972 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:03.355981    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:19:03.368938 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:03.368914    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:19:03.766250 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:03.766216    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:19:10.907955 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:10.907920    2584 scope.go:117] "RemoveContainer" containerID="d7f8e3724930b3d24a498b9276e885172e872c942d54dc4f71c9e4700e06221b"
Jul 19 17:19:14.839372 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.839340    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"]
Jul 19 17:19:14.844138 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.844117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:14.846443 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.846419    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ec4179\"/\"kube-root-ca.crt\""
Jul 19 17:19:14.847581 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.847560    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ec4179\"/\"default-dockercfg-4jhmd\""
Jul 19 17:19:14.847683 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.847560    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ec4179\"/\"openshift-service-ca.crt\""
Jul 19 17:19:14.847683 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.847646    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ec4179\"/\"test-model-ec4179-kserve-self-signed-certs\""
Jul 19 17:19:14.851423 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.851400    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"]
Jul 19 17:19:14.856495 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.856464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:14.937732 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.937692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tls-certs\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:14.937928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.937743    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgff9\" (UniqueName: \"kubernetes.io/projected/7f83d96f-a2ca-4dcb-822e-47f6512baba8-kube-api-access-qgff9\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:14.937928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.937803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-model-cache\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:14.937928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.937828    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-home\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:14.937928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.937843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tmp-dir\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:14.937928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:14.937873    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-dshm\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039241 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-dshm\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039478 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tls-certs\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039478 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039331    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgff9\" (UniqueName: \"kubernetes.io/projected/7f83d96f-a2ca-4dcb-822e-47f6512baba8-kube-api-access-qgff9\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039478 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039406    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-model-cache\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039478 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039453    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-home\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039478 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039473    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tmp-dir\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039810 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039778    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-model-cache\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039878 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039815    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tmp-dir\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.039878 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.039850    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-home\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.041691 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.041658    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-dshm\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.041923 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.041908    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tls-certs\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.047112 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.047092    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgff9\" (UniqueName: \"kubernetes.io/projected/7f83d96f-a2ca-4dcb-822e-47f6512baba8-kube-api-access-qgff9\") pod \"test-model-ec4179-kserve-6cdbd8599-8t28g\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") " pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.155690 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.155658    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:15.297120 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.297094    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"]
Jul 19 17:19:15.299426 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:19:15.299396    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f83d96f_a2ca_4dcb_822e_47f6512baba8.slice/crio-42116a4b0e66f5071464bbb470f3db7c085349c41835c4474755177fd612c8a4 WatchSource:0}: Error finding container 42116a4b0e66f5071464bbb470f3db7c085349c41835c4474755177fd612c8a4: Status 404 returned error can't find the container with id 42116a4b0e66f5071464bbb470f3db7c085349c41835c4474755177fd612c8a4
Jul 19 17:19:15.816956 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.816916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g" event={"ID":"7f83d96f-a2ca-4dcb-822e-47f6512baba8","Type":"ContainerStarted","Data":"3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4"}
Jul 19 17:19:15.816956 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.816964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g" event={"ID":"7f83d96f-a2ca-4dcb-822e-47f6512baba8","Type":"ContainerStarted","Data":"42116a4b0e66f5071464bbb470f3db7c085349c41835c4474755177fd612c8a4"}
Jul 19 17:19:15.833135 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:15.833081    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g" podStartSLOduration=1.8330640630000001 podStartE2EDuration="1.833064063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:19:15.831991499 +0000 UTC m=+2465.737242922" watchObservedRunningTime="2026-07-19 17:19:15.833064063 +0000 UTC m=+2465.738315487"
Jul 19 17:19:18.002153 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:18.002119    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:18.102399 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:18.102356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:25.156163 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:25.156119    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:25.158714 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:25.156175    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:25.169492 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:25.169464    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:25.878753 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:25.878720    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:34.734452 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:34.733389    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"]
Jul 19 17:19:34.754866 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:34.754837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:35.940354 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:35.940308    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"]
Jul 19 17:19:35.961702 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:35.961672    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:36.061747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.061707    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"]
Jul 19 17:19:36.065260 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.065234    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:36.067643 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.067621    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:19:36.071732 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.071709    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"]
Jul 19 17:19:36.133603 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.133499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65fa455-a0b0-478b-ba28-267b3a2a5805-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz\" (UID: \"d65fa455-a0b0-478b-ba28-267b3a2a5805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:36.234544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.234444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65fa455-a0b0-478b-ba28-267b3a2a5805-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz\" (UID: \"d65fa455-a0b0-478b-ba28-267b3a2a5805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:36.235075 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.235053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65fa455-a0b0-478b-ba28-267b3a2a5805-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz\" (UID: \"d65fa455-a0b0-478b-ba28-267b3a2a5805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:36.380679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.380644    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:36.521859 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.521796    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"]
Jul 19 17:19:36.524369 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:19:36.524336    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65fa455_a0b0_478b_ba28_267b3a2a5805.slice/crio-20e924241e3c713f6daab8ede352295fb9b0d14b938b632ef76ed5a2d38f6291 WatchSource:0}: Error finding container 20e924241e3c713f6daab8ede352295fb9b0d14b938b632ef76ed5a2d38f6291: Status 404 returned error can't find the container with id 20e924241e3c713f6daab8ede352295fb9b0d14b938b632ef76ed5a2d38f6291
Jul 19 17:19:36.556409 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.556378    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:36.656476 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.656435    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:36.923380 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.923343    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz" event={"ID":"d65fa455-a0b0-478b-ba28-267b3a2a5805","Type":"ContainerStarted","Data":"767c92ef47364e789529974b5626589a9210684553c1841288ac09ca23def92e"}
Jul 19 17:19:36.923559 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.923393    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz" event={"ID":"d65fa455-a0b0-478b-ba28-267b3a2a5805","Type":"ContainerStarted","Data":"20e924241e3c713f6daab8ede352295fb9b0d14b938b632ef76ed5a2d38f6291"}
Jul 19 17:19:36.938892 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:36.938846    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz" podStartSLOduration=0.938831826 podStartE2EDuration="938.831826ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:19:36.937001045 +0000 UTC m=+2486.842252470" watchObservedRunningTime="2026-07-19 17:19:36.938831826 +0000 UTC m=+2486.844083250"
Jul 19 17:19:37.931401 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:37.931366    2584 generic.go:356] "Generic (PLEG): container finished" podID="d65fa455-a0b0-478b-ba28-267b3a2a5805" containerID="767c92ef47364e789529974b5626589a9210684553c1841288ac09ca23def92e" exitCode=0
Jul 19 17:19:37.931849 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:37.931459    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz" event={"ID":"d65fa455-a0b0-478b-ba28-267b3a2a5805","Type":"ContainerDied","Data":"767c92ef47364e789529974b5626589a9210684553c1841288ac09ca23def92e"}
Jul 19 17:19:39.078920 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.078893    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:39.158501 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.158466    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65fa455-a0b0-478b-ba28-267b3a2a5805-maas-api-service-ca\") pod \"d65fa455-a0b0-478b-ba28-267b3a2a5805\" (UID: \"d65fa455-a0b0-478b-ba28-267b3a2a5805\") "
Jul 19 17:19:39.158846 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.158822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65fa455-a0b0-478b-ba28-267b3a2a5805-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d65fa455-a0b0-478b-ba28-267b3a2a5805" (UID: "d65fa455-a0b0-478b-ba28-267b3a2a5805"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:19:39.259311 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.259202    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65fa455-a0b0-478b-ba28-267b3a2a5805-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:39.943439 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.943404    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"
Jul 19 17:19:39.943624 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.943442    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz" event={"ID":"d65fa455-a0b0-478b-ba28-267b3a2a5805","Type":"ContainerDied","Data":"20e924241e3c713f6daab8ede352295fb9b0d14b938b632ef76ed5a2d38f6291"}
Jul 19 17:19:39.943624 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:39.943494    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e924241e3c713f6daab8ede352295fb9b0d14b938b632ef76ed5a2d38f6291"
Jul 19 17:19:46.426323 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.426266    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"]
Jul 19 17:19:46.426845 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.426631    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8" podUID="6d98552d-ac48-4da0-a801-6466f816a5bf" containerName="maas-api" containerID="cri-o://71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5" gracePeriod=30
Jul 19 17:19:46.455487 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.455456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:46.700928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.700900    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:19:46.733603 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.733571    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d98552d-ac48-4da0-a801-6466f816a5bf-openshift-service-ca\") pod \"6d98552d-ac48-4da0-a801-6466f816a5bf\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") "
Jul 19 17:19:46.733791 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.733613    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch2dn\" (UniqueName: \"kubernetes.io/projected/6d98552d-ac48-4da0-a801-6466f816a5bf-kube-api-access-ch2dn\") pod \"6d98552d-ac48-4da0-a801-6466f816a5bf\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") "
Jul 19 17:19:46.733791 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.733661    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d98552d-ac48-4da0-a801-6466f816a5bf-maas-api-tls\") pod \"6d98552d-ac48-4da0-a801-6466f816a5bf\" (UID: \"6d98552d-ac48-4da0-a801-6466f816a5bf\") "
Jul 19 17:19:46.733970 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.733942    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d98552d-ac48-4da0-a801-6466f816a5bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d98552d-ac48-4da0-a801-6466f816a5bf" (UID: "6d98552d-ac48-4da0-a801-6466f816a5bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:19:46.736011 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.735976    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d98552d-ac48-4da0-a801-6466f816a5bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d98552d-ac48-4da0-a801-6466f816a5bf" (UID: "6d98552d-ac48-4da0-a801-6466f816a5bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:19:46.736109 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.736062    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d98552d-ac48-4da0-a801-6466f816a5bf-kube-api-access-ch2dn" (OuterVolumeSpecName: "kube-api-access-ch2dn") pod "6d98552d-ac48-4da0-a801-6466f816a5bf" (UID: "6d98552d-ac48-4da0-a801-6466f816a5bf"). InnerVolumeSpecName "kube-api-access-ch2dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:19:46.835400 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.835361    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d98552d-ac48-4da0-a801-6466f816a5bf-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:46.835400 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.835393    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch2dn\" (UniqueName: \"kubernetes.io/projected/6d98552d-ac48-4da0-a801-6466f816a5bf-kube-api-access-ch2dn\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:46.835400 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.835404    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d98552d-ac48-4da0-a801-6466f816a5bf-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:46.981036 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.980931    2584 generic.go:356] "Generic (PLEG): container finished" podID="6d98552d-ac48-4da0-a801-6466f816a5bf" containerID="71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5" exitCode=0
Jul 19 17:19:46.981036 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.980986    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"
Jul 19 17:19:46.981228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.981024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8" event={"ID":"6d98552d-ac48-4da0-a801-6466f816a5bf","Type":"ContainerDied","Data":"71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5"}
Jul 19 17:19:46.981228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.981086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8" event={"ID":"6d98552d-ac48-4da0-a801-6466f816a5bf","Type":"ContainerDied","Data":"657089a9c8cbce31730b5b238badeba9aee6f73afde3243bd454ea8a02bb7241"}
Jul 19 17:19:46.981228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.981118    2584 scope.go:117] "RemoveContainer" containerID="71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5"
Jul 19 17:19:46.991380 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.991359    2584 scope.go:117] "RemoveContainer" containerID="71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5"
Jul 19 17:19:46.991654 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:19:46.991637    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5\": container with ID starting with 71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5 not found: ID does not exist" containerID="71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5"
Jul 19 17:19:46.991719 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:46.991662    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5"} err="failed to get container status \"71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5\": rpc error: code = NotFound desc = could not find container \"71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5\": container with ID starting with 71dabe9d23edae813ae68a90155074d3d96a705184f88d5976fa119af0fb58a5 not found: ID does not exist"
Jul 19 17:19:47.002738 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:47.002711    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"]
Jul 19 17:19:47.005727 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:47.005703    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7dceef-697b4c8bb6-dvzq8"]
Jul 19 17:19:48.632615 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:48.632580    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d98552d-ac48-4da0-a801-6466f816a5bf" path="/var/lib/kubelet/pods/6d98552d-ac48-4da0-a801-6466f816a5bf/volumes"
Jul 19 17:19:49.756230 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:49.756188    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl" podUID="a6112f29-0888-4b85-b187-f9b3f8718162" containerName="main" containerID="cri-o://105bd6ff7ffcef106a9a61fe42866d96e95980e8ff4f76fbee5f82d77567cd70" gracePeriod=45
Jul 19 17:19:49.998236 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:49.998202    2584 generic.go:356] "Generic (PLEG): container finished" podID="a6112f29-0888-4b85-b187-f9b3f8718162" containerID="105bd6ff7ffcef106a9a61fe42866d96e95980e8ff4f76fbee5f82d77567cd70" exitCode=0
Jul 19 17:19:49.998448 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:49.998269    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl" event={"ID":"a6112f29-0888-4b85-b187-f9b3f8718162","Type":"ContainerDied","Data":"105bd6ff7ffcef106a9a61fe42866d96e95980e8ff4f76fbee5f82d77567cd70"}
Jul 19 17:19:50.037980 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.037956    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:19:50.167043 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-dshm\") pod \"a6112f29-0888-4b85-b187-f9b3f8718162\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") "
Jul 19 17:19:50.167252 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167075    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-home\") pod \"a6112f29-0888-4b85-b187-f9b3f8718162\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") "
Jul 19 17:19:50.167252 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167116    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6112f29-0888-4b85-b187-f9b3f8718162-tls-certs\") pod \"a6112f29-0888-4b85-b187-f9b3f8718162\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") "
Jul 19 17:19:50.167252 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167149    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l842\" (UniqueName: \"kubernetes.io/projected/a6112f29-0888-4b85-b187-f9b3f8718162-kube-api-access-9l842\") pod \"a6112f29-0888-4b85-b187-f9b3f8718162\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") "
Jul 19 17:19:50.167252 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167207    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-model-cache\") pod \"a6112f29-0888-4b85-b187-f9b3f8718162\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") "
Jul 19 17:19:50.167492 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167256    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-tmp-dir\") pod \"a6112f29-0888-4b85-b187-f9b3f8718162\" (UID: \"a6112f29-0888-4b85-b187-f9b3f8718162\") "
Jul 19 17:19:50.167492 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167373    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-home" (OuterVolumeSpecName: "home") pod "a6112f29-0888-4b85-b187-f9b3f8718162" (UID: "a6112f29-0888-4b85-b187-f9b3f8718162"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:50.167583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167506    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6112f29-0888-4b85-b187-f9b3f8718162" (UID: "a6112f29-0888-4b85-b187-f9b3f8718162"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:50.167583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167567    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6112f29-0888-4b85-b187-f9b3f8718162" (UID: "a6112f29-0888-4b85-b187-f9b3f8718162"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:50.167685 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167590    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:50.167685 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.167601    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:50.169495 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.169466    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-dshm" (OuterVolumeSpecName: "dshm") pod "a6112f29-0888-4b85-b187-f9b3f8718162" (UID: "a6112f29-0888-4b85-b187-f9b3f8718162"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:50.169610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.169503    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6112f29-0888-4b85-b187-f9b3f8718162-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6112f29-0888-4b85-b187-f9b3f8718162" (UID: "a6112f29-0888-4b85-b187-f9b3f8718162"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:19:50.169610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.169540    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6112f29-0888-4b85-b187-f9b3f8718162-kube-api-access-9l842" (OuterVolumeSpecName: "kube-api-access-9l842") pod "a6112f29-0888-4b85-b187-f9b3f8718162" (UID: "a6112f29-0888-4b85-b187-f9b3f8718162"). InnerVolumeSpecName "kube-api-access-9l842". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:19:50.268983 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.268886    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:50.268983 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.268921    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6112f29-0888-4b85-b187-f9b3f8718162-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:50.268983 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.268933    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6112f29-0888-4b85-b187-f9b3f8718162-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:50.268983 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.268946    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l842\" (UniqueName: \"kubernetes.io/projected/a6112f29-0888-4b85-b187-f9b3f8718162-kube-api-access-9l842\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:50.963538 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:50.963496    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g" podUID="7f83d96f-a2ca-4dcb-822e-47f6512baba8" containerName="main" containerID="cri-o://3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4" gracePeriod=45
Jul 19 17:19:51.007022 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.006993    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"
Jul 19 17:19:51.007022 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.007005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl" event={"ID":"a6112f29-0888-4b85-b187-f9b3f8718162","Type":"ContainerDied","Data":"8f1611b83672a8b87cac6120f80db78808dfb2d83e2ef204006f1bbf78efc460"}
Jul 19 17:19:51.007327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.007053    2584 scope.go:117] "RemoveContainer" containerID="105bd6ff7ffcef106a9a61fe42866d96e95980e8ff4f76fbee5f82d77567cd70"
Jul 19 17:19:51.037764 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.037732    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"]
Jul 19 17:19:51.039581 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.039559    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7dceef/test-model-7dceef-kserve-57549bbb65-h7hrl"]
Jul 19 17:19:51.130577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.130543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:51.140538 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.140368    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"]
Jul 19 17:19:51.143884 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.143855    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7dceef-6r6xz"]
Jul 19 17:19:51.228777 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.228692    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:51.277151 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277118    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-dshm\") pod \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") "
Jul 19 17:19:51.277349 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277179    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgff9\" (UniqueName: \"kubernetes.io/projected/7f83d96f-a2ca-4dcb-822e-47f6512baba8-kube-api-access-qgff9\") pod \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") "
Jul 19 17:19:51.277349 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277251    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-home\") pod \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") "
Jul 19 17:19:51.277494 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277352    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tmp-dir\") pod \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") "
Jul 19 17:19:51.277494 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277388    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-model-cache\") pod \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") "
Jul 19 17:19:51.277494 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277443    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tls-certs\") pod \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\" (UID: \"7f83d96f-a2ca-4dcb-822e-47f6512baba8\") "
Jul 19 17:19:51.277627 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277551    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-home" (OuterVolumeSpecName: "home") pod "7f83d96f-a2ca-4dcb-822e-47f6512baba8" (UID: "7f83d96f-a2ca-4dcb-822e-47f6512baba8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:51.277733 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277688    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f83d96f-a2ca-4dcb-822e-47f6512baba8" (UID: "7f83d96f-a2ca-4dcb-822e-47f6512baba8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:51.277859 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277837    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:51.278019 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277998    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:51.278019 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.277928    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f83d96f-a2ca-4dcb-822e-47f6512baba8" (UID: "7f83d96f-a2ca-4dcb-822e-47f6512baba8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:51.280117 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.280095    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-dshm" (OuterVolumeSpecName: "dshm") pod "7f83d96f-a2ca-4dcb-822e-47f6512baba8" (UID: "7f83d96f-a2ca-4dcb-822e-47f6512baba8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:19:51.280293 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.280258    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f83d96f-a2ca-4dcb-822e-47f6512baba8" (UID: "7f83d96f-a2ca-4dcb-822e-47f6512baba8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:19:51.280369 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.280264    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f83d96f-a2ca-4dcb-822e-47f6512baba8-kube-api-access-qgff9" (OuterVolumeSpecName: "kube-api-access-qgff9") pod "7f83d96f-a2ca-4dcb-822e-47f6512baba8" (UID: "7f83d96f-a2ca-4dcb-822e-47f6512baba8"). InnerVolumeSpecName "kube-api-access-qgff9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:19:51.378915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.378878    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:51.378915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.378908    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f83d96f-a2ca-4dcb-822e-47f6512baba8-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:51.378915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.378917    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f83d96f-a2ca-4dcb-822e-47f6512baba8-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:51.378915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.378926    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgff9\" (UniqueName: \"kubernetes.io/projected/7f83d96f-a2ca-4dcb-822e-47f6512baba8-kube-api-access-qgff9\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:19:51.494259 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:51.494179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:52.014107 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.014069    2584 generic.go:356] "Generic (PLEG): container finished" podID="7f83d96f-a2ca-4dcb-822e-47f6512baba8" containerID="3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4" exitCode=0
Jul 19 17:19:52.014107 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.014113    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"
Jul 19 17:19:52.014651 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.014158    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g" event={"ID":"7f83d96f-a2ca-4dcb-822e-47f6512baba8","Type":"ContainerDied","Data":"3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4"}
Jul 19 17:19:52.014651 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.014200    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g" event={"ID":"7f83d96f-a2ca-4dcb-822e-47f6512baba8","Type":"ContainerDied","Data":"42116a4b0e66f5071464bbb470f3db7c085349c41835c4474755177fd612c8a4"}
Jul 19 17:19:52.014651 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.014217    2584 scope.go:117] "RemoveContainer" containerID="3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4"
Jul 19 17:19:52.023730 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.023711    2584 scope.go:117] "RemoveContainer" containerID="3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4"
Jul 19 17:19:52.023982 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:19:52.023963    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4\": container with ID starting with 3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4 not found: ID does not exist" containerID="3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4"
Jul 19 17:19:52.024062 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.023994    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4"} err="failed to get container status \"3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4\": rpc error: code = NotFound desc = could not find container \"3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4\": container with ID starting with 3429a4e3489dbccc3a3ddc6848bc09006ea9163c6d7c0e3b1bf1d808a9ce97d4 not found: ID does not exist"
Jul 19 17:19:52.040440 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.040408    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"]
Jul 19 17:19:52.042112 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.042092    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ec4179/test-model-ec4179-kserve-6cdbd8599-8t28g"]
Jul 19 17:19:52.633746 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.633707    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f83d96f-a2ca-4dcb-822e-47f6512baba8" path="/var/lib/kubelet/pods/7f83d96f-a2ca-4dcb-822e-47f6512baba8/volumes"
Jul 19 17:19:52.634161 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.634145    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6112f29-0888-4b85-b187-f9b3f8718162" path="/var/lib/kubelet/pods/a6112f29-0888-4b85-b187-f9b3f8718162/volumes"
Jul 19 17:19:52.634502 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:52.634488    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d65fa455-a0b0-478b-ba28-267b3a2a5805" path="/var/lib/kubelet/pods/d65fa455-a0b0-478b-ba28-267b3a2a5805/volumes"
Jul 19 17:19:57.280037 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.280004    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:57.348115 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.348079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:57.562789 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.562688    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:57.802566 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.802533    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:57.924185 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924144    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"]
Jul 19 17:19:57.924630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924610    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f83d96f-a2ca-4dcb-822e-47f6512baba8" containerName="main"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924643    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f83d96f-a2ca-4dcb-822e-47f6512baba8" containerName="main"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924657    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d65fa455-a0b0-478b-ba28-267b3a2a5805" containerName="revoke-keys"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924665    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d65fa455-a0b0-478b-ba28-267b3a2a5805" containerName="revoke-keys"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924686    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6112f29-0888-4b85-b187-f9b3f8718162" containerName="main"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924695    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6112f29-0888-4b85-b187-f9b3f8718162" containerName="main"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924716    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d98552d-ac48-4da0-a801-6466f816a5bf" containerName="maas-api"
Jul 19 17:19:57.924742 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924723    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d98552d-ac48-4da0-a801-6466f816a5bf" containerName="maas-api"
Jul 19 17:19:57.925076 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924819    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f83d96f-a2ca-4dcb-822e-47f6512baba8" containerName="main"
Jul 19 17:19:57.925076 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924835    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d98552d-ac48-4da0-a801-6466f816a5bf" containerName="maas-api"
Jul 19 17:19:57.925076 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924844    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d65fa455-a0b0-478b-ba28-267b3a2a5805" containerName="revoke-keys"
Jul 19 17:19:57.925076 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.924853    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6112f29-0888-4b85-b187-f9b3f8718162" containerName="main"
Jul 19 17:19:57.929600 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.929577    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:19:57.932251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.932225    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:19:57.934594 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:57.934566    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"]
Jul 19 17:19:58.041683 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.041646    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7da18-1647-4758-a8c4-b6bffc10fc92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv\" (UID: \"68e7da18-1647-4758-a8c4-b6bffc10fc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:19:58.142456 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.142412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7da18-1647-4758-a8c4-b6bffc10fc92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv\" (UID: \"68e7da18-1647-4758-a8c4-b6bffc10fc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:19:58.143075 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.143055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7da18-1647-4758-a8c4-b6bffc10fc92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv\" (UID: \"68e7da18-1647-4758-a8c4-b6bffc10fc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:19:58.242382 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.242297    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:19:58.385315 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.385269    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"]
Jul 19 17:19:58.387774 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:19:58.387743    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e7da18_1647_4758_a8c4_b6bffc10fc92.slice/crio-061f06c51d16d1e12acafebdac2174f2bf90cf6a6d23b0a090be6552b2ce90c6 WatchSource:0}: Error finding container 061f06c51d16d1e12acafebdac2174f2bf90cf6a6d23b0a090be6552b2ce90c6: Status 404 returned error can't find the container with id 061f06c51d16d1e12acafebdac2174f2bf90cf6a6d23b0a090be6552b2ce90c6
Jul 19 17:19:58.450755 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.450706    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:58.551881 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:58.551767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:19:59.053148 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:59.053103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv" event={"ID":"68e7da18-1647-4758-a8c4-b6bffc10fc92","Type":"ContainerStarted","Data":"6f268541986e1896dcffd0cf2878904ad883a2683dafcaf838d4993854044ad8"}
Jul 19 17:19:59.053329 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:59.053165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv" event={"ID":"68e7da18-1647-4758-a8c4-b6bffc10fc92","Type":"ContainerStarted","Data":"061f06c51d16d1e12acafebdac2174f2bf90cf6a6d23b0a090be6552b2ce90c6"}
Jul 19 17:19:59.069514 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:19:59.069462    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv" podStartSLOduration=2.069446856 podStartE2EDuration="2.069446856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:19:59.067450865 +0000 UTC m=+2508.972702288" watchObservedRunningTime="2026-07-19 17:19:59.069446856 +0000 UTC m=+2508.974698279"
Jul 19 17:20:00.060545 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:00.060506    2584 generic.go:356] "Generic (PLEG): container finished" podID="68e7da18-1647-4758-a8c4-b6bffc10fc92" containerID="6f268541986e1896dcffd0cf2878904ad883a2683dafcaf838d4993854044ad8" exitCode=0
Jul 19 17:20:00.060932 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:00.060590    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv" event={"ID":"68e7da18-1647-4758-a8c4-b6bffc10fc92","Type":"ContainerDied","Data":"6f268541986e1896dcffd0cf2878904ad883a2683dafcaf838d4993854044ad8"}
Jul 19 17:20:01.206765 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:01.206739    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:20:01.271684 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:01.271653    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7da18-1647-4758-a8c4-b6bffc10fc92-maas-api-service-ca\") pod \"68e7da18-1647-4758-a8c4-b6bffc10fc92\" (UID: \"68e7da18-1647-4758-a8c4-b6bffc10fc92\") "
Jul 19 17:20:01.272038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:01.272010    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e7da18-1647-4758-a8c4-b6bffc10fc92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68e7da18-1647-4758-a8c4-b6bffc10fc92" (UID: "68e7da18-1647-4758-a8c4-b6bffc10fc92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:20:01.373055 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:01.372960    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7da18-1647-4758-a8c4-b6bffc10fc92-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:01.751786 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:01.751712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:02.072110 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:02.072020    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"
Jul 19 17:20:02.072110 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:02.072024    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv" event={"ID":"68e7da18-1647-4758-a8c4-b6bffc10fc92","Type":"ContainerDied","Data":"061f06c51d16d1e12acafebdac2174f2bf90cf6a6d23b0a090be6552b2ce90c6"}
Jul 19 17:20:02.072110 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:02.072097    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="061f06c51d16d1e12acafebdac2174f2bf90cf6a6d23b0a090be6552b2ce90c6"
Jul 19 17:20:05.659145 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:05.659110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:05.759572 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:05.759536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:08.287635 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.287602    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"]
Jul 19 17:20:08.288073 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.287818    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" podUID="c5f0c8b1-e8ec-4e96-8c5c-29971c581963" containerName="maas-api" containerID="cri-o://01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900" gracePeriod=30
Jul 19 17:20:08.312604 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.312570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:08.742361 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.742337    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:20:08.840438 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.840356    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls\") pod \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") "
Jul 19 17:20:08.840438 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.840395    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snmwd\" (UniqueName: \"kubernetes.io/projected/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-kube-api-access-snmwd\") pod \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") "
Jul 19 17:20:08.840624 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.840491    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-openshift-service-ca\") pod \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\" (UID: \"c5f0c8b1-e8ec-4e96-8c5c-29971c581963\") "
Jul 19 17:20:08.840822 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.840789    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5f0c8b1-e8ec-4e96-8c5c-29971c581963" (UID: "c5f0c8b1-e8ec-4e96-8c5c-29971c581963"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:20:08.842654 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.842631    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-kube-api-access-snmwd" (OuterVolumeSpecName: "kube-api-access-snmwd") pod "c5f0c8b1-e8ec-4e96-8c5c-29971c581963" (UID: "c5f0c8b1-e8ec-4e96-8c5c-29971c581963"). InnerVolumeSpecName "kube-api-access-snmwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:20:08.842654 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.842632    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5f0c8b1-e8ec-4e96-8c5c-29971c581963" (UID: "c5f0c8b1-e8ec-4e96-8c5c-29971c581963"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:20:08.941304 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.941247    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:08.941304 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.941299    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:08.941304 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:08.941309    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snmwd\" (UniqueName: \"kubernetes.io/projected/c5f0c8b1-e8ec-4e96-8c5c-29971c581963-kube-api-access-snmwd\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:09.107420 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.107321    2584 generic.go:356] "Generic (PLEG): container finished" podID="c5f0c8b1-e8ec-4e96-8c5c-29971c581963" containerID="01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900" exitCode=0
Jul 19 17:20:09.107420 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.107384    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"
Jul 19 17:20:09.107420 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.107404    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" event={"ID":"c5f0c8b1-e8ec-4e96-8c5c-29971c581963","Type":"ContainerDied","Data":"01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900"}
Jul 19 17:20:09.107653 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.107482    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n" event={"ID":"c5f0c8b1-e8ec-4e96-8c5c-29971c581963","Type":"ContainerDied","Data":"a190e53f7b1065d80412ff1ec187208cd391c5c7953e377d9e753d4634a688a4"}
Jul 19 17:20:09.107653 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.107506    2584 scope.go:117] "RemoveContainer" containerID="01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900"
Jul 19 17:20:09.117376 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.117356    2584 scope.go:117] "RemoveContainer" containerID="01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900"
Jul 19 17:20:09.117640 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:20:09.117619    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900\": container with ID starting with 01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900 not found: ID does not exist" containerID="01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900"
Jul 19 17:20:09.117687 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.117650    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900"} err="failed to get container status \"01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900\": rpc error: code = NotFound desc = could not find container \"01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900\": container with ID starting with 01437c5a1093c5142c746ed863b9d5b7fb6e93af1de10ef6725b541965f8b900 not found: ID does not exist"
Jul 19 17:20:09.128147 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.128115    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"]
Jul 19 17:20:09.130150 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:09.130127    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ec4179-7bdc64cd66-mq88n"]
Jul 19 17:20:10.633572 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:10.633542    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f0c8b1-e8ec-4e96-8c5c-29971c581963" path="/var/lib/kubelet/pods/c5f0c8b1-e8ec-4e96-8c5c-29971c581963/volumes"
Jul 19 17:20:12.989393 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:12.989364    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"]
Jul 19 17:20:12.992464 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:12.992439    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ec4179-ptvdv"]
Jul 19 17:20:12.994040 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:12.994012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:13.386643 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:13.386562    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:14.632787 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:14.632749    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68e7da18-1647-4758-a8c4-b6bffc10fc92" path="/var/lib/kubelet/pods/68e7da18-1647-4758-a8c4-b6bffc10fc92/volumes"
Jul 19 17:20:18.946754 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:18.946714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:19.002683 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:19.002644    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:19.212149 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:19.212061    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:19.439339 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:19.439306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:19.780916 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:19.780882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:20.094700 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:20.094605    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:20.212893 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:20.212831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:22.802451 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.802414    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:22.894672 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.894566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.896975    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"]
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.897819    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68e7da18-1647-4758-a8c4-b6bffc10fc92" containerName="revoke-keys"
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.897839    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="68e7da18-1647-4758-a8c4-b6bffc10fc92" containerName="revoke-keys"
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.897853    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f0c8b1-e8ec-4e96-8c5c-29971c581963" containerName="maas-api"
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.897862    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f0c8b1-e8ec-4e96-8c5c-29971c581963" containerName="maas-api"
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.898035    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f0c8b1-e8ec-4e96-8c5c-29971c581963" containerName="maas-api"
Jul 19 17:20:22.900133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.898062    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="68e7da18-1647-4758-a8c4-b6bffc10fc92" containerName="revoke-keys"
Jul 19 17:20:22.904856 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.904833    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:22.907815 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.907792    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5c6a59-serving-cert\""
Jul 19 17:20:22.908188 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.908124    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"]
Jul 19 17:20:22.960616 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.960582    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:22.967869 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.967839    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:22.968039 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.967901    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4ld\" (UniqueName: \"kubernetes.io/projected/ca5801fa-779d-428c-a29f-e62e69b67ec8-kube-api-access-tl4ld\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:22.968039 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:22.967932    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5801fa-779d-428c-a29f-e62e69b67ec8-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.069066 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.068973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.069228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.069079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4ld\" (UniqueName: \"kubernetes.io/projected/ca5801fa-779d-428c-a29f-e62e69b67ec8-kube-api-access-tl4ld\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.069228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.069127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5801fa-779d-428c-a29f-e62e69b67ec8-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.069228 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:20:23.069144    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-serving-cert: secret "maas-api-e2e-api-a-5c6a59-serving-cert" not found
Jul 19 17:20:23.069228 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:20:23.069212    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls podName:ca5801fa-779d-428c-a29f-e62e69b67ec8 nodeName:}" failed. No retries permitted until 2026-07-19 17:20:23.569189854 +0000 UTC m=+2533.474441274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls") pod "maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" (UID: "ca5801fa-779d-428c-a29f-e62e69b67ec8") : secret "maas-api-e2e-api-a-5c6a59-serving-cert" not found
Jul 19 17:20:23.069831 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.069811    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5801fa-779d-428c-a29f-e62e69b67ec8-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.078905 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.078878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4ld\" (UniqueName: \"kubernetes.io/projected/ca5801fa-779d-428c-a29f-e62e69b67ec8-kube-api-access-tl4ld\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.165433 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.165398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:23.351191 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.351093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:23.451660 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.451631    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:23.572436 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.572398    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.574950 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.574924    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls\") pod \"maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.819613 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.819571    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:23.968697 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:23.968670    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"]
Jul 19 17:20:23.971107 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:20:23.971081    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5801fa_779d_428c_a29f_e62e69b67ec8.slice/crio-6c065310523e988c7fb9ce23cbda8402d9a1b6e29464fece863739407594afca WatchSource:0}: Error finding container 6c065310523e988c7fb9ce23cbda8402d9a1b6e29464fece863739407594afca: Status 404 returned error can't find the container with id 6c065310523e988c7fb9ce23cbda8402d9a1b6e29464fece863739407594afca
Jul 19 17:20:24.184957 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:24.184916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" event={"ID":"ca5801fa-779d-428c-a29f-e62e69b67ec8","Type":"ContainerStarted","Data":"6c065310523e988c7fb9ce23cbda8402d9a1b6e29464fece863739407594afca"}
Jul 19 17:20:25.192045 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:25.192009    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" event={"ID":"ca5801fa-779d-428c-a29f-e62e69b67ec8","Type":"ContainerStarted","Data":"bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a"}
Jul 19 17:20:25.192454 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:25.192089    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:25.209541 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:25.209488    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" podStartSLOduration=2.802331903 podStartE2EDuration="3.209471931s" totalImagesPullingTime="407.140028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:20:25.208316618 +0000 UTC m=+2535.113568042" watchObservedRunningTime="2026-07-19 17:20:25.209471931 +0000 UTC m=+2535.114723355"
Jul 19 17:20:27.238385 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:27.238353    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:27.682934 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:27.682901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:28.007255 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:28.007141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:28.192851 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:28.192816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:30.736194 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.736147    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:30.840331 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.840267    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:30.924631 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.924601    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"]
Jul 19 17:20:30.928412 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.928388    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:30.931113 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.931087    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-35dde0-serving-cert\""
Jul 19 17:20:30.939633 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.939603    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"]
Jul 19 17:20:30.963825 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:30.963792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:31.034424 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.034264    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.034424 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.034399    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.034627 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.034439    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbkb\" (UniqueName: \"kubernetes.io/projected/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-kube-api-access-7hbkb\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.135467 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.135415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.135467 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.135481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.135753 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:20:31.135591    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-serving-cert: secret "maas-api-e2e-api-b-35dde0-serving-cert" not found
Jul 19 17:20:31.135753 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.135649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbkb\" (UniqueName: \"kubernetes.io/projected/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-kube-api-access-7hbkb\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.135753 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:20:31.135664    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls podName:7add51ad-a3ed-4388-b012-7a4beb0fd5b4 nodeName:}" failed. No retries permitted until 2026-07-19 17:20:31.635643323 +0000 UTC m=+2541.540894726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls") pod "maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" (UID: "7add51ad-a3ed-4388-b012-7a4beb0fd5b4") : secret "maas-api-e2e-api-b-35dde0-serving-cert" not found
Jul 19 17:20:31.136135 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.136118    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-openshift-service-ca\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.144398 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.144368    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbkb\" (UniqueName: \"kubernetes.io/projected/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-kube-api-access-7hbkb\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.204228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.204202    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:31.306984 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.306905    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:31.417138 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.417107    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:31.517656 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.517620    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:31.641318 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.641195    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.643884 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.643859    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls\") pod \"maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.840757 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.840722    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:31.985982 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.985951    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"]
Jul 19 17:20:31.987245 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:20:31.987221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7add51ad_a3ed_4388_b012_7a4beb0fd5b4.slice/crio-1df9a5aec1668b0d41b241c8f6b713692e538ddef2101c9e3ebf129b51582af5 WatchSource:0}: Error finding container 1df9a5aec1668b0d41b241c8f6b713692e538ddef2101c9e3ebf129b51582af5: Status 404 returned error can't find the container with id 1df9a5aec1668b0d41b241c8f6b713692e538ddef2101c9e3ebf129b51582af5
Jul 19 17:20:31.988852 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:31.988833    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 17:20:32.228726 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:32.228687    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" event={"ID":"7add51ad-a3ed-4388-b012-7a4beb0fd5b4","Type":"ContainerStarted","Data":"1df9a5aec1668b0d41b241c8f6b713692e538ddef2101c9e3ebf129b51582af5"}
Jul 19 17:20:33.235133 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:33.235093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" event={"ID":"7add51ad-a3ed-4388-b012-7a4beb0fd5b4","Type":"ContainerStarted","Data":"31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8"}
Jul 19 17:20:33.235577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:33.235231    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:33.252854 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:33.252812    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" podStartSLOduration=2.8765284920000003 podStartE2EDuration="3.252796767s" totalImagesPullingTime="376.268275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:20:33.252326455 +0000 UTC m=+2543.157577881" watchObservedRunningTime="2026-07-19 17:20:33.252796767 +0000 UTC m=+2543.158048190"
Jul 19 17:20:35.237583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:35.237546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:39.245678 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:39.245645    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:20:43.356988 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.356944    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"]
Jul 19 17:20:43.360603 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.360580    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:43.362875 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.362851    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:20:43.367255 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.367112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"]
Jul 19 17:20:43.449079 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.449033    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14778-3973-4c3e-a616-4bb3700764c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2\" (UID: \"d1c14778-3973-4c3e-a616-4bb3700764c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:43.550225 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.550183    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14778-3973-4c3e-a616-4bb3700764c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2\" (UID: \"d1c14778-3973-4c3e-a616-4bb3700764c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:43.550967 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.550940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14778-3973-4c3e-a616-4bb3700764c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2\" (UID: \"d1c14778-3973-4c3e-a616-4bb3700764c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:43.673307 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.673254    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:43.817647 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:43.817603    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"]
Jul 19 17:20:43.819855 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:20:43.819820    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c14778_3973_4c3e_a616_4bb3700764c1.slice/crio-5c3e0564ea8fd1e8657570eb86d9b4b5e343adb5c9248542b71c5656da0c1cc5 WatchSource:0}: Error finding container 5c3e0564ea8fd1e8657570eb86d9b4b5e343adb5c9248542b71c5656da0c1cc5: Status 404 returned error can't find the container with id 5c3e0564ea8fd1e8657570eb86d9b4b5e343adb5c9248542b71c5656da0c1cc5
Jul 19 17:20:44.044658 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:44.044622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:44.145237 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:44.145207    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:44.296015 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:44.295919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2" event={"ID":"d1c14778-3973-4c3e-a616-4bb3700764c1","Type":"ContainerStarted","Data":"321bcdf40afa60e90f66e4459acc9c27670e37220b877acdbbdb14813d04b5f1"}
Jul 19 17:20:44.296015 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:44.295985    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2" event={"ID":"d1c14778-3973-4c3e-a616-4bb3700764c1","Type":"ContainerStarted","Data":"5c3e0564ea8fd1e8657570eb86d9b4b5e343adb5c9248542b71c5656da0c1cc5"}
Jul 19 17:20:44.311107 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:44.311048    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2" podStartSLOduration=1.311029201 podStartE2EDuration="1.311029201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:20:44.308985517 +0000 UTC m=+2554.214236937" watchObservedRunningTime="2026-07-19 17:20:44.311029201 +0000 UTC m=+2554.216280625"
Jul 19 17:20:45.303040 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:45.303000    2584 generic.go:356] "Generic (PLEG): container finished" podID="d1c14778-3973-4c3e-a616-4bb3700764c1" containerID="321bcdf40afa60e90f66e4459acc9c27670e37220b877acdbbdb14813d04b5f1" exitCode=0
Jul 19 17:20:45.303454 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:45.303092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2" event={"ID":"d1c14778-3973-4c3e-a616-4bb3700764c1","Type":"ContainerDied","Data":"321bcdf40afa60e90f66e4459acc9c27670e37220b877acdbbdb14813d04b5f1"}
Jul 19 17:20:46.450317 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:46.450268    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:46.577747 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:46.577649    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14778-3973-4c3e-a616-4bb3700764c1-maas-api-service-ca\") pod \"d1c14778-3973-4c3e-a616-4bb3700764c1\" (UID: \"d1c14778-3973-4c3e-a616-4bb3700764c1\") "
Jul 19 17:20:46.578038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:46.578013    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c14778-3973-4c3e-a616-4bb3700764c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1c14778-3973-4c3e-a616-4bb3700764c1" (UID: "d1c14778-3973-4c3e-a616-4bb3700764c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:20:46.679170 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:46.678944    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c14778-3973-4c3e-a616-4bb3700764c1-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:47.315118 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:47.315083    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2" event={"ID":"d1c14778-3973-4c3e-a616-4bb3700764c1","Type":"ContainerDied","Data":"5c3e0564ea8fd1e8657570eb86d9b4b5e343adb5c9248542b71c5656da0c1cc5"}
Jul 19 17:20:47.315332 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:47.315133    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3e0564ea8fd1e8657570eb86d9b4b5e343adb5c9248542b71c5656da0c1cc5"
Jul 19 17:20:47.315332 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:47.315097    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"
Jul 19 17:20:52.228964 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:52.228929    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:52.330006 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:52.329972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:53.858823 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:53.858786    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:53.960891 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:53.960858    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:59.086567 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.086528    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"]
Jul 19 17:20:59.087106 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.086796    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" podUID="ca5801fa-779d-428c-a29f-e62e69b67ec8" containerName="maas-api" containerID="cri-o://bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a" gracePeriod=30
Jul 19 17:20:59.110955 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.110927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:20:59.346735 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.346672    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:59.376421 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.376388    2584 generic.go:356] "Generic (PLEG): container finished" podID="ca5801fa-779d-428c-a29f-e62e69b67ec8" containerID="bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a" exitCode=0
Jul 19 17:20:59.376601 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.376441    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"
Jul 19 17:20:59.376601 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.376471    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" event={"ID":"ca5801fa-779d-428c-a29f-e62e69b67ec8","Type":"ContainerDied","Data":"bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a"}
Jul 19 17:20:59.376601 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.376528    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx" event={"ID":"ca5801fa-779d-428c-a29f-e62e69b67ec8","Type":"ContainerDied","Data":"6c065310523e988c7fb9ce23cbda8402d9a1b6e29464fece863739407594afca"}
Jul 19 17:20:59.376601 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.376553    2584 scope.go:117] "RemoveContainer" containerID="bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a"
Jul 19 17:20:59.388146 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.388124    2584 scope.go:117] "RemoveContainer" containerID="bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a"
Jul 19 17:20:59.388516 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:20:59.388489    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a\": container with ID starting with bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a not found: ID does not exist" containerID="bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a"
Jul 19 17:20:59.388610 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.388533    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a"} err="failed to get container status \"bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a\": rpc error: code = NotFound desc = could not find container \"bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a\": container with ID starting with bd309b1b12ea5b6cbae024f3df3c61cd2395f796ad284b20540046e7843bc86a not found: ID does not exist"
Jul 19 17:20:59.490673 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.490639    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5801fa-779d-428c-a29f-e62e69b67ec8-openshift-service-ca\") pod \"ca5801fa-779d-428c-a29f-e62e69b67ec8\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") "
Jul 19 17:20:59.490863 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.490689    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls\") pod \"ca5801fa-779d-428c-a29f-e62e69b67ec8\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") "
Jul 19 17:20:59.490863 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.490713    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl4ld\" (UniqueName: \"kubernetes.io/projected/ca5801fa-779d-428c-a29f-e62e69b67ec8-kube-api-access-tl4ld\") pod \"ca5801fa-779d-428c-a29f-e62e69b67ec8\" (UID: \"ca5801fa-779d-428c-a29f-e62e69b67ec8\") "
Jul 19 17:20:59.491108 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.491076    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5801fa-779d-428c-a29f-e62e69b67ec8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca5801fa-779d-428c-a29f-e62e69b67ec8" (UID: "ca5801fa-779d-428c-a29f-e62e69b67ec8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:20:59.493008 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.492979    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca5801fa-779d-428c-a29f-e62e69b67ec8" (UID: "ca5801fa-779d-428c-a29f-e62e69b67ec8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:20:59.493118 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.492980    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5801fa-779d-428c-a29f-e62e69b67ec8-kube-api-access-tl4ld" (OuterVolumeSpecName: "kube-api-access-tl4ld") pod "ca5801fa-779d-428c-a29f-e62e69b67ec8" (UID: "ca5801fa-779d-428c-a29f-e62e69b67ec8"). InnerVolumeSpecName "kube-api-access-tl4ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:20:59.591557 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.591513    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca5801fa-779d-428c-a29f-e62e69b67ec8-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:59.591557 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.591553    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca5801fa-779d-428c-a29f-e62e69b67ec8-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:59.591761 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.591569    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl4ld\" (UniqueName: \"kubernetes.io/projected/ca5801fa-779d-428c-a29f-e62e69b67ec8-kube-api-access-tl4ld\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:20:59.700130 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.700095    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"]
Jul 19 17:20:59.703914 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:20:59.703889    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c6a59-7c6f7ff964-7lxxx"]
Jul 19 17:21:00.633199 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:00.633166    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5801fa-779d-428c-a29f-e62e69b67ec8" path="/var/lib/kubelet/pods/ca5801fa-779d-428c-a29f-e62e69b67ec8/volumes"
Jul 19 17:21:03.425527 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:03.425488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:03.430605 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:03.430560    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"]
Jul 19 17:21:03.433045 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:03.433023    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6a59-sfxj2"]
Jul 19 17:21:03.803885 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:03.803799    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:04.633265 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:04.633217    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c14778-3973-4c3e-a616-4bb3700764c1" path="/var/lib/kubelet/pods/d1c14778-3973-4c3e-a616-4bb3700764c1/volumes"
Jul 19 17:21:09.610416 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:09.610379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:09.710480 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:09.710439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:09.897744 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:09.897659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:10.133176 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.133141    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:10.268336 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268301    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"]
Jul 19 17:21:10.268691 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268677    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5801fa-779d-428c-a29f-e62e69b67ec8" containerName="maas-api"
Jul 19 17:21:10.268735 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268692    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5801fa-779d-428c-a29f-e62e69b67ec8" containerName="maas-api"
Jul 19 17:21:10.268735 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268710    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c14778-3973-4c3e-a616-4bb3700764c1" containerName="revoke-keys"
Jul 19 17:21:10.268735 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268715    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c14778-3973-4c3e-a616-4bb3700764c1" containerName="revoke-keys"
Jul 19 17:21:10.268832 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268771    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c14778-3973-4c3e-a616-4bb3700764c1" containerName="revoke-keys"
Jul 19 17:21:10.268832 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.268781    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5801fa-779d-428c-a29f-e62e69b67ec8" containerName="maas-api"
Jul 19 17:21:10.273140 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.273118    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:10.275871 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.275849    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:21:10.280396 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.280370    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"]
Jul 19 17:21:10.292430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.292395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c51014-dd54-4f3c-ae35-ddb2112f8733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-35dde0-cbp96\" (UID: \"24c51014-dd54-4f3c-ae35-ddb2112f8733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:10.392997 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.392959    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c51014-dd54-4f3c-ae35-ddb2112f8733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-35dde0-cbp96\" (UID: \"24c51014-dd54-4f3c-ae35-ddb2112f8733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:10.393622 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.393598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c51014-dd54-4f3c-ae35-ddb2112f8733-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-35dde0-cbp96\" (UID: \"24c51014-dd54-4f3c-ae35-ddb2112f8733\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:10.589922 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.589841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:21:10.598157 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.598129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:10.738215 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.738191    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"]
Jul 19 17:21:10.740038 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:21:10.740011    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c51014_dd54_4f3c_ae35_ddb2112f8733.slice/crio-16eebe1e2059ae1f61421fadbb6567110880f6c047a4b0cedee3f88c9a2f84a0 WatchSource:0}: Error finding container 16eebe1e2059ae1f61421fadbb6567110880f6c047a4b0cedee3f88c9a2f84a0: Status 404 returned error can't find the container with id 16eebe1e2059ae1f61421fadbb6567110880f6c047a4b0cedee3f88c9a2f84a0
Jul 19 17:21:10.867821 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.867718    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:10.950996 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.950966    2584 scope.go:117] "RemoveContainer" containerID="68468d747e5e8caf2df92e99a247befb0dc45082c522213efa4c4c0611c920e4"
Jul 19 17:21:10.969863 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:10.969839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:11.441065 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:11.441023    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96" event={"ID":"24c51014-dd54-4f3c-ae35-ddb2112f8733","Type":"ContainerStarted","Data":"83fd6aa04f2ab821f7b2a4618fe95366cabbef51eb42902c6a1f089ddc5e90b4"}
Jul 19 17:21:11.441234 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:11.441090    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96" event={"ID":"24c51014-dd54-4f3c-ae35-ddb2112f8733","Type":"ContainerStarted","Data":"16eebe1e2059ae1f61421fadbb6567110880f6c047a4b0cedee3f88c9a2f84a0"}
Jul 19 17:21:11.455684 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:11.455638    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96" podStartSLOduration=1.455598141 podStartE2EDuration="1.455598141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:21:11.454944559 +0000 UTC m=+2581.360195983" watchObservedRunningTime="2026-07-19 17:21:11.455598141 +0000 UTC m=+2581.360849566"
Jul 19 17:21:12.447033 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:12.446994    2584 generic.go:356] "Generic (PLEG): container finished" podID="24c51014-dd54-4f3c-ae35-ddb2112f8733" containerID="83fd6aa04f2ab821f7b2a4618fe95366cabbef51eb42902c6a1f089ddc5e90b4" exitCode=0
Jul 19 17:21:12.447474 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:12.447084    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96" event={"ID":"24c51014-dd54-4f3c-ae35-ddb2112f8733","Type":"ContainerDied","Data":"83fd6aa04f2ab821f7b2a4618fe95366cabbef51eb42902c6a1f089ddc5e90b4"}
Jul 19 17:21:13.591736 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:13.591712    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:13.623978 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:13.623954    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c51014-dd54-4f3c-ae35-ddb2112f8733-maas-api-service-ca\") pod \"24c51014-dd54-4f3c-ae35-ddb2112f8733\" (UID: \"24c51014-dd54-4f3c-ae35-ddb2112f8733\") "
Jul 19 17:21:13.624338 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:13.624318    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c51014-dd54-4f3c-ae35-ddb2112f8733-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24c51014-dd54-4f3c-ae35-ddb2112f8733" (UID: "24c51014-dd54-4f3c-ae35-ddb2112f8733"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:21:13.725265 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:13.725176    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c51014-dd54-4f3c-ae35-ddb2112f8733-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:21:14.458597 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:14.458565    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"
Jul 19 17:21:14.458782 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:14.458569    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96" event={"ID":"24c51014-dd54-4f3c-ae35-ddb2112f8733","Type":"ContainerDied","Data":"16eebe1e2059ae1f61421fadbb6567110880f6c047a4b0cedee3f88c9a2f84a0"}
Jul 19 17:21:14.458782 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:14.458697    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16eebe1e2059ae1f61421fadbb6567110880f6c047a4b0cedee3f88c9a2f84a0"
Jul 19 17:21:25.663704 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:25.663598    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"]
Jul 19 17:21:25.664271 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:25.663851    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" podUID="7add51ad-a3ed-4388-b012-7a4beb0fd5b4" containerName="maas-api" containerID="cri-o://31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8" gracePeriod=30
Jul 19 17:21:25.686343 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:25.686311    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:25.922167 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:25.922102    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:21:26.038987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.038950    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls\") pod \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") "
Jul 19 17:21:26.039165 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.039023    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-openshift-service-ca\") pod \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") "
Jul 19 17:21:26.039165 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.039049    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hbkb\" (UniqueName: \"kubernetes.io/projected/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-kube-api-access-7hbkb\") pod \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\" (UID: \"7add51ad-a3ed-4388-b012-7a4beb0fd5b4\") "
Jul 19 17:21:26.039409 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.039385    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7add51ad-a3ed-4388-b012-7a4beb0fd5b4" (UID: "7add51ad-a3ed-4388-b012-7a4beb0fd5b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:21:26.041430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.041401    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7add51ad-a3ed-4388-b012-7a4beb0fd5b4" (UID: "7add51ad-a3ed-4388-b012-7a4beb0fd5b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:21:26.041546 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.041433    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-kube-api-access-7hbkb" (OuterVolumeSpecName: "kube-api-access-7hbkb") pod "7add51ad-a3ed-4388-b012-7a4beb0fd5b4" (UID: "7add51ad-a3ed-4388-b012-7a4beb0fd5b4"). InnerVolumeSpecName "kube-api-access-7hbkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:21:26.140586 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.140548    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:21:26.140586 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.140578    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:21:26.140586 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.140589    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbkb\" (UniqueName: \"kubernetes.io/projected/7add51ad-a3ed-4388-b012-7a4beb0fd5b4-kube-api-access-7hbkb\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:21:26.522211 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.522177    2584 generic.go:356] "Generic (PLEG): container finished" podID="7add51ad-a3ed-4388-b012-7a4beb0fd5b4" containerID="31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8" exitCode=0
Jul 19 17:21:26.522445 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.522222    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"
Jul 19 17:21:26.522445 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.522263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" event={"ID":"7add51ad-a3ed-4388-b012-7a4beb0fd5b4","Type":"ContainerDied","Data":"31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8"}
Jul 19 17:21:26.522445 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.522350    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v" event={"ID":"7add51ad-a3ed-4388-b012-7a4beb0fd5b4","Type":"ContainerDied","Data":"1df9a5aec1668b0d41b241c8f6b713692e538ddef2101c9e3ebf129b51582af5"}
Jul 19 17:21:26.522445 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.522382    2584 scope.go:117] "RemoveContainer" containerID="31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8"
Jul 19 17:21:26.531532 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.531515    2584 scope.go:117] "RemoveContainer" containerID="31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8"
Jul 19 17:21:26.531791 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:21:26.531770    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8\": container with ID starting with 31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8 not found: ID does not exist" containerID="31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8"
Jul 19 17:21:26.531832 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.531799    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8"} err="failed to get container status \"31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8\": rpc error: code = NotFound desc = could not find container \"31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8\": container with ID starting with 31d29a020c0badfe739de6954259f4f3c6ef1ee927b4964236c7b99244a447c8 not found: ID does not exist"
Jul 19 17:21:26.542880 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.542857    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"]
Jul 19 17:21:26.545951 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.545930    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-35dde0-78489d46fc-7gg9v"]
Jul 19 17:21:26.633021 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:26.632989    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7add51ad-a3ed-4388-b012-7a4beb0fd5b4" path="/var/lib/kubelet/pods/7add51ad-a3ed-4388-b012-7a4beb0fd5b4/volumes"
Jul 19 17:21:30.344657 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:30.344614    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:30.348738 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:30.348710    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"]
Jul 19 17:21:30.350907 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:30.350881    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-35dde0-cbp96"]
Jul 19 17:21:30.632585 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:30.632501    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c51014-dd54-4f3c-ae35-ddb2112f8733" path="/var/lib/kubelet/pods/24c51014-dd54-4f3c-ae35-ddb2112f8733/volumes"
Jul 19 17:21:30.746165 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:30.746133    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:36.359624 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:36.359536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:36.420485 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:36.420450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:36.637661 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:36.637568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:36.856853 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:36.856814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:37.221855 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:37.221818    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:37.538358 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:37.538256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:37.672836 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:37.672802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:40.095661 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.095616    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:40.184344 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.184306    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"]
Jul 19 17:21:40.184889 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.184866    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7add51ad-a3ed-4388-b012-7a4beb0fd5b4" containerName="maas-api"
Jul 19 17:21:40.185069 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.185047    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7add51ad-a3ed-4388-b012-7a4beb0fd5b4" containerName="maas-api"
Jul 19 17:21:40.185201 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.185076    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c51014-dd54-4f3c-ae35-ddb2112f8733" containerName="revoke-keys"
Jul 19 17:21:40.185201 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.185086    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c51014-dd54-4f3c-ae35-ddb2112f8733" containerName="revoke-keys"
Jul 19 17:21:40.185364 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.185226    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7add51ad-a3ed-4388-b012-7a4beb0fd5b4" containerName="maas-api"
Jul 19 17:21:40.185364 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.185239    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c51014-dd54-4f3c-ae35-ddb2112f8733" containerName="revoke-keys"
Jul 19 17:21:40.186333 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.186301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:40.190103 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.190077    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.193003 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.192986    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-8d2863-serving-cert\""
Jul 19 17:21:40.199038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.199014    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"]
Jul 19 17:21:40.254014 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.253980    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:40.265105 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.265069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.265105 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.265108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b962c19-0a4c-44bf-8f97-e7525a8f513a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.265368 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.265133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l758q\" (UniqueName: \"kubernetes.io/projected/3b962c19-0a4c-44bf-8f97-e7525a8f513a-kube-api-access-l758q\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.366269 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.366171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.366269 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.366208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b962c19-0a4c-44bf-8f97-e7525a8f513a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.366269 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.366235    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l758q\" (UniqueName: \"kubernetes.io/projected/3b962c19-0a4c-44bf-8f97-e7525a8f513a-kube-api-access-l758q\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.366603 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:21:40.366374    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-serving-cert: secret "maas-api-e2e-shared-a-8d2863-serving-cert" not found
Jul 19 17:21:40.366603 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:21:40.366471    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls podName:3b962c19-0a4c-44bf-8f97-e7525a8f513a nodeName:}" failed. No retries permitted until 2026-07-19 17:21:40.866448275 +0000 UTC m=+2610.771699691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls") pod "maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" (UID: "3b962c19-0a4c-44bf-8f97-e7525a8f513a") : secret "maas-api-e2e-shared-a-8d2863-serving-cert" not found
Jul 19 17:21:40.366914 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.366894    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b962c19-0a4c-44bf-8f97-e7525a8f513a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.375442 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.375417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l758q\" (UniqueName: \"kubernetes.io/projected/3b962c19-0a4c-44bf-8f97-e7525a8f513a-kube-api-access-l758q\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.438893 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.438850    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:40.844271 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.844233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:40.871388 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.871359    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.873903 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.873872    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls\") pod \"maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:40.946180 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:40.946139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:41.104425 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:41.104329    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:41.254319 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:41.254272    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"]
Jul 19 17:21:41.258461 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:21:41.258433    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b962c19_0a4c_44bf_8f97_e7525a8f513a.slice/crio-f3db33ed7d1fbaa89c01c2283c6607000503790cfb20f99295438ec1a0b010eb WatchSource:0}: Error finding container f3db33ed7d1fbaa89c01c2283c6607000503790cfb20f99295438ec1a0b010eb: Status 404 returned error can't find the container with id f3db33ed7d1fbaa89c01c2283c6607000503790cfb20f99295438ec1a0b010eb
Jul 19 17:21:41.602688 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:41.602652    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" event={"ID":"3b962c19-0a4c-44bf-8f97-e7525a8f513a","Type":"ContainerStarted","Data":"f3db33ed7d1fbaa89c01c2283c6607000503790cfb20f99295438ec1a0b010eb"}
Jul 19 17:21:42.609990 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:42.609949    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" event={"ID":"3b962c19-0a4c-44bf-8f97-e7525a8f513a","Type":"ContainerStarted","Data":"9aba380302518ee3c1b22d0ce1507728f04f1bdf9119d963a3dcbe39308fce03"}
Jul 19 17:21:42.610428 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:42.610069    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:42.625967 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:42.625896    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" podStartSLOduration=2.252509861 podStartE2EDuration="2.625879852s" totalImagesPullingTime="373.369991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:21:42.625061679 +0000 UTC m=+2612.530313103" watchObservedRunningTime="2026-07-19 17:21:42.625879852 +0000 UTC m=+2612.531131276"
Jul 19 17:21:44.633104 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:44.633072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:44.970803 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:44.970766    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:45.285674 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:45.285580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:45.356592 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:45.356551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:45.458188 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:45.458139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:45.907959 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:45.907925    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:48.138352 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.138314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:48.234094 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.234063    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:48.236408 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.236375    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"]
Jul 19 17:21:48.240501 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.240477    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.243271 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.243247    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-0fa4bd-serving-cert\""
Jul 19 17:21:48.251038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.251012    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"]
Jul 19 17:21:48.306759 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.306696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:48.344980 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.344940    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942884a3-1bc5-4546-9987-e0c627765e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.345175 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.344989    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znck8\" (UniqueName: \"kubernetes.io/projected/942884a3-1bc5-4546-9987-e0c627765e4b-kube-api-access-znck8\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.345175 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.345058    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942884a3-1bc5-4546-9987-e0c627765e4b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.445728 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.445685    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942884a3-1bc5-4546-9987-e0c627765e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.445914 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.445736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znck8\" (UniqueName: \"kubernetes.io/projected/942884a3-1bc5-4546-9987-e0c627765e4b-kube-api-access-znck8\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.445914 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.445784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942884a3-1bc5-4546-9987-e0c627765e4b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.446614 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.446507    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942884a3-1bc5-4546-9987-e0c627765e4b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.448498 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.448469    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942884a3-1bc5-4546-9987-e0c627765e4b-maas-api-tls\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.457702 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.457675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:48.461953 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.461932    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znck8\" (UniqueName: \"kubernetes.io/projected/942884a3-1bc5-4546-9987-e0c627765e4b-kube-api-access-znck8\") pod \"maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.553408 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.553371    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:48.622074 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.622046    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:21:48.706828 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.706500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"]
Jul 19 17:21:48.708312 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:21:48.708262    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942884a3_1bc5_4546_9987_e0c627765e4b.slice/crio-2a9e2a7a864f5546b1a5febbc0a546a2e56d452ac3e04bb74b94b51013511598 WatchSource:0}: Error finding container 2a9e2a7a864f5546b1a5febbc0a546a2e56d452ac3e04bb74b94b51013511598: Status 404 returned error can't find the container with id 2a9e2a7a864f5546b1a5febbc0a546a2e56d452ac3e04bb74b94b51013511598
Jul 19 17:21:48.898153 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:48.898116    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:49.000664 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:49.000577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:49.656380 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:49.656346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" event={"ID":"942884a3-1bc5-4546-9987-e0c627765e4b","Type":"ContainerStarted","Data":"ba992f7197945927702ab17ee13342c0e3920f216121713cc11aeaaadd843c7e"}
Jul 19 17:21:49.656807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:49.656395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" event={"ID":"942884a3-1bc5-4546-9987-e0c627765e4b","Type":"ContainerStarted","Data":"2a9e2a7a864f5546b1a5febbc0a546a2e56d452ac3e04bb74b94b51013511598"}
Jul 19 17:21:49.656807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:49.656446    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:21:49.672660 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:49.672599    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" podStartSLOduration=1.316810405 podStartE2EDuration="1.672580717s" totalImagesPullingTime="355.770312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:21:49.671237684 +0000 UTC m=+2619.576489109" watchObservedRunningTime="2026-07-19 17:21:49.672580717 +0000 UTC m=+2619.577832159"
Jul 19 17:21:52.687698 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:52.687663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:21:55.667611 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:21:55.667570    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:22:01.569562 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:01.569504    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:01.670552 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:01.670517    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:06.524152 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.524115    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"]
Jul 19 17:22:06.528027 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.528007    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.530714 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.530692    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8d2863\"/\"openshift-service-ca.crt\""
Jul 19 17:22:06.531885 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.531860    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8d2863\"/\"auth-test-model-8d2863-kserve-self-signed-certs\""
Jul 19 17:22:06.532003 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.531900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8d2863\"/\"default-dockercfg-plnb5\""
Jul 19 17:22:06.532069 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.532020    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8d2863\"/\"kube-root-ca.crt\""
Jul 19 17:22:06.536544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.536359    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"]
Jul 19 17:22:06.545498 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.545470    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:06.606202 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.606168    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-home\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.606202 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.606205    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-model-cache\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.606486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.606250    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-dshm\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.606486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.606384    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f7403-1368-4814-9a54-7ba842a225f0-tls-certs\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.606486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.606417    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-tmp-dir\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.606486 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.606444    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfz8\" (UniqueName: \"kubernetes.io/projected/9b5f7403-1368-4814-9a54-7ba842a225f0-kube-api-access-9hfz8\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.707847 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.707802    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-dshm\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.707881    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f7403-1368-4814-9a54-7ba842a225f0-tls-certs\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.707917    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-tmp-dir\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.707935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfz8\" (UniqueName: \"kubernetes.io/projected/9b5f7403-1368-4814-9a54-7ba842a225f0-kube-api-access-9hfz8\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.707961    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-home\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.707981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-model-cache\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708390 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.708362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-home\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708512 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.708402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-tmp-dir\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.708512 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.708478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-model-cache\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.710376 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.710347    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-dshm\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.710622 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.710605    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f7403-1368-4814-9a54-7ba842a225f0-tls-certs\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.715246 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.715225    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfz8\" (UniqueName: \"kubernetes.io/projected/9b5f7403-1368-4814-9a54-7ba842a225f0-kube-api-access-9hfz8\") pod \"auth-test-model-8d2863-kserve-57897cb4c-vw4gf\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") " pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.839924 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.839828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:06.982068 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:06.982039    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"]
Jul 19 17:22:06.984623 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:22:06.984591    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b5f7403_1368_4814_9a54_7ba842a225f0.slice/crio-cb0967b403986cbb484226a2b3a3fa006df7f2ad5e24abfebe6a924a170b6c52 WatchSource:0}: Error finding container cb0967b403986cbb484226a2b3a3fa006df7f2ad5e24abfebe6a924a170b6c52: Status 404 returned error can't find the container with id cb0967b403986cbb484226a2b3a3fa006df7f2ad5e24abfebe6a924a170b6c52
Jul 19 17:22:07.757945 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:07.757912    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf" event={"ID":"9b5f7403-1368-4814-9a54-7ba842a225f0","Type":"ContainerStarted","Data":"c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97"}
Jul 19 17:22:07.758367 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:07.757953    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf" event={"ID":"9b5f7403-1368-4814-9a54-7ba842a225f0","Type":"ContainerStarted","Data":"cb0967b403986cbb484226a2b3a3fa006df7f2ad5e24abfebe6a924a170b6c52"}
Jul 19 17:22:07.775026 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:07.774978    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf" podStartSLOduration=1.7749594279999998 podStartE2EDuration="1.774959428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:22:07.772940724 +0000 UTC m=+2637.678192149" watchObservedRunningTime="2026-07-19 17:22:07.774959428 +0000 UTC m=+2637.680210902"
Jul 19 17:22:10.474760 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:10.474719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:10.579460 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:10.579424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:10.975221 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:10.975179    2584 scope.go:117] "RemoveContainer" containerID="c86a59230800c1e4005479405a3c5505a5309e05658dd9b515a7ddc7f9cf0b04"
Jul 19 17:22:16.840726 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:16.840690    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:16.841152 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:16.840751    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:16.856558 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:16.856527    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:17.822902 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:17.822872    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:22:27.409164 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:27.409129    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:27.511680 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:27.511649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:28.748339 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.748305    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"]
Jul 19 17:22:28.751989 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.751970    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.754494 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.754477    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"openshift-service-ca.crt\""
Jul 19 17:22:28.755617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.755576    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"kube-root-ca.crt\""
Jul 19 17:22:28.755617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.755590    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"default-dockercfg-5z74t\""
Jul 19 17:22:28.755765 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.755614    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"auth-test-model-0fa4bd-kserve-self-signed-certs\""
Jul 19 17:22:28.759728 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.759411    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"]
Jul 19 17:22:28.816005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.815650    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-model-cache\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.816005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.815694    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-tmp-dir\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.816005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.815724    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f36157-1532-47e0-94a1-e884f0986386-tls-certs\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.816005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.815770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-dshm\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.816005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.815816    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-home\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.816005 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.815874    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj88w\" (UniqueName: \"kubernetes.io/projected/52f36157-1532-47e0-94a1-e884f0986386-kube-api-access-fj88w\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.837503 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.837476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:22:28.916341 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj88w\" (UniqueName: \"kubernetes.io/projected/52f36157-1532-47e0-94a1-e884f0986386-kube-api-access-fj88w\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916393    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-model-cache\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916420    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-tmp-dir\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f36157-1532-47e0-94a1-e884f0986386-tls-certs\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916482    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-dshm\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916544 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916528    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-home\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916903 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916878    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-model-cache\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.916980 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-tmp-dir\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.917021 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.916986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-home\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.919052 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.919024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-dshm\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.919258 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.919243    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f36157-1532-47e0-94a1-e884f0986386-tls-certs\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:28.924415 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:28.924393    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj88w\" (UniqueName: \"kubernetes.io/projected/52f36157-1532-47e0-94a1-e884f0986386-kube-api-access-fj88w\") pod \"auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") " pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:29.063624 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:29.063535    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:29.205038 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:29.205002    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"]
Jul 19 17:22:29.208358 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:22:29.208329    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52f36157_1532_47e0_94a1_e884f0986386.slice/crio-7ebd7699534d6ba6901bd63b7a01767119f6a033573c1658afe41523ce9625ec WatchSource:0}: Error finding container 7ebd7699534d6ba6901bd63b7a01767119f6a033573c1658afe41523ce9625ec: Status 404 returned error can't find the container with id 7ebd7699534d6ba6901bd63b7a01767119f6a033573c1658afe41523ce9625ec
Jul 19 17:22:29.876131 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:29.876092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr" event={"ID":"52f36157-1532-47e0-94a1-e884f0986386","Type":"ContainerStarted","Data":"d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252"}
Jul 19 17:22:29.876584 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:29.876141    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr" event={"ID":"52f36157-1532-47e0-94a1-e884f0986386","Type":"ContainerStarted","Data":"7ebd7699534d6ba6901bd63b7a01767119f6a033573c1658afe41523ce9625ec"}
Jul 19 17:22:29.894811 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:29.894764    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr" podStartSLOduration=1.894746739 podStartE2EDuration="1.894746739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:22:29.893221717 +0000 UTC m=+2659.798473142" watchObservedRunningTime="2026-07-19 17:22:29.894746739 +0000 UTC m=+2659.799998200"
Jul 19 17:22:39.064554 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:39.064515    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:39.064993 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:39.064592    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:39.076818 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:39.076792    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:22:39.939712 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:22:39.939684    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:23:10.758196 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:23:10.758113    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:23:10.764404 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:23:10.764384    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:24:40.937423 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:40.937346    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"]
Jul 19 17:24:40.991100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:40.991070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:24:42.122198 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:42.122159    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"]
Jul 19 17:24:42.134652 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:42.134622    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:24:54.739378 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.739341    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"]
Jul 19 17:24:54.742752 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.742732    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.745193 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.745171    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8d2863\"/\"sub-test-model-8d2863-kserve-self-signed-certs\""
Jul 19 17:24:54.749630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.749580    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"]
Jul 19 17:24:54.833913 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.833875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:24:54.883150 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.883108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhrfh\" (UniqueName: \"kubernetes.io/projected/958303f3-e33a-4904-919f-6973276c43af-kube-api-access-nhrfh\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.883414 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.883216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958303f3-e33a-4904-919f-6973276c43af-tls-certs\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.883414 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.883245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-model-cache\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.883530 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.883466    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-home\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.883530 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.883501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-dshm\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.883684 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.883555    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-tmp-dir\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.984941 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.984902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958303f3-e33a-4904-919f-6973276c43af-tls-certs\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.984941 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.984941    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-model-cache\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985184 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-home\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985240 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-dshm\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985334 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985270    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-tmp-dir\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985391 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985341    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrfh\" (UniqueName: \"kubernetes.io/projected/958303f3-e33a-4904-919f-6973276c43af-kube-api-access-nhrfh\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985456 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-model-cache\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985552 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-home\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.985615 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.985597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-tmp-dir\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.987731 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.987709    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958303f3-e33a-4904-919f-6973276c43af-tls-certs\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.987863 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.987790    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-dshm\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:54.993748 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:54.993691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrfh\" (UniqueName: \"kubernetes.io/projected/958303f3-e33a-4904-919f-6973276c43af-kube-api-access-nhrfh\") pod \"sub-test-model-8d2863-kserve-d9dc7556f-m9mlq\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") " pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:55.054810 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:55.054769    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:24:55.197973 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:55.197948    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"]
Jul 19 17:24:55.200039 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:24:55.200010    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod958303f3_e33a_4904_919f_6973276c43af.slice/crio-c4d262da76fe56ffb9f21858e708ce321a72662293ac6b5ec3ed96f53134fc20 WatchSource:0}: Error finding container c4d262da76fe56ffb9f21858e708ce321a72662293ac6b5ec3ed96f53134fc20: Status 404 returned error can't find the container with id c4d262da76fe56ffb9f21858e708ce321a72662293ac6b5ec3ed96f53134fc20
Jul 19 17:24:55.644781 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:55.644695    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq" event={"ID":"958303f3-e33a-4904-919f-6973276c43af","Type":"ContainerStarted","Data":"01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73"}
Jul 19 17:24:55.644781 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:55.644732    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq" event={"ID":"958303f3-e33a-4904-919f-6973276c43af","Type":"ContainerStarted","Data":"c4d262da76fe56ffb9f21858e708ce321a72662293ac6b5ec3ed96f53134fc20"}
Jul 19 17:24:55.662647 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:55.662594    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq" podStartSLOduration=1.6625752299999998 podStartE2EDuration="1.66257523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:24:55.660908035 +0000 UTC m=+2805.566159460" watchObservedRunningTime="2026-07-19 17:24:55.66257523 +0000 UTC m=+2805.567826655"
Jul 19 17:24:55.959218 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:55.959169    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf" podUID="9b5f7403-1368-4814-9a54-7ba842a225f0" containerName="main" containerID="cri-o://c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97" gracePeriod=45
Jul 19 17:24:56.222315 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.222236    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:24:56.399473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399440    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-tmp-dir\") pod \"9b5f7403-1368-4814-9a54-7ba842a225f0\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") "
Jul 19 17:24:56.399769 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399500    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f7403-1368-4814-9a54-7ba842a225f0-tls-certs\") pod \"9b5f7403-1368-4814-9a54-7ba842a225f0\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") "
Jul 19 17:24:56.399769 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399546    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-dshm\") pod \"9b5f7403-1368-4814-9a54-7ba842a225f0\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") "
Jul 19 17:24:56.399769 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399565    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-home\") pod \"9b5f7403-1368-4814-9a54-7ba842a225f0\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") "
Jul 19 17:24:56.399769 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399620    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-model-cache\") pod \"9b5f7403-1368-4814-9a54-7ba842a225f0\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") "
Jul 19 17:24:56.399769 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399653    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hfz8\" (UniqueName: \"kubernetes.io/projected/9b5f7403-1368-4814-9a54-7ba842a225f0-kube-api-access-9hfz8\") pod \"9b5f7403-1368-4814-9a54-7ba842a225f0\" (UID: \"9b5f7403-1368-4814-9a54-7ba842a225f0\") "
Jul 19 17:24:56.400071 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399767    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b5f7403-1368-4814-9a54-7ba842a225f0" (UID: "9b5f7403-1368-4814-9a54-7ba842a225f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:56.400071 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399862    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-home" (OuterVolumeSpecName: "home") pod "9b5f7403-1368-4814-9a54-7ba842a225f0" (UID: "9b5f7403-1368-4814-9a54-7ba842a225f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:56.400071 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.399908    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b5f7403-1368-4814-9a54-7ba842a225f0" (UID: "9b5f7403-1368-4814-9a54-7ba842a225f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:56.400435 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.400070    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:56.400435 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.400089    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:56.400435 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.400098    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:56.401965 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.401929    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5f7403-1368-4814-9a54-7ba842a225f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b5f7403-1368-4814-9a54-7ba842a225f0" (UID: "9b5f7403-1368-4814-9a54-7ba842a225f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:24:56.401965 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.401949    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-dshm" (OuterVolumeSpecName: "dshm") pod "9b5f7403-1368-4814-9a54-7ba842a225f0" (UID: "9b5f7403-1368-4814-9a54-7ba842a225f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:56.402107 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.402032    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5f7403-1368-4814-9a54-7ba842a225f0-kube-api-access-9hfz8" (OuterVolumeSpecName: "kube-api-access-9hfz8") pod "9b5f7403-1368-4814-9a54-7ba842a225f0" (UID: "9b5f7403-1368-4814-9a54-7ba842a225f0"). InnerVolumeSpecName "kube-api-access-9hfz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:24:56.500981 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.500895    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f7403-1368-4814-9a54-7ba842a225f0-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:56.500981 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.500927    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b5f7403-1368-4814-9a54-7ba842a225f0-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:56.500981 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.500938    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hfz8\" (UniqueName: \"kubernetes.io/projected/9b5f7403-1368-4814-9a54-7ba842a225f0-kube-api-access-9hfz8\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:56.651071 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.651037    2584 generic.go:356] "Generic (PLEG): container finished" podID="9b5f7403-1368-4814-9a54-7ba842a225f0" containerID="c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97" exitCode=0
Jul 19 17:24:56.651260 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.651083    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"
Jul 19 17:24:56.651260 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.651117    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf" event={"ID":"9b5f7403-1368-4814-9a54-7ba842a225f0","Type":"ContainerDied","Data":"c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97"}
Jul 19 17:24:56.651260 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.651173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf" event={"ID":"9b5f7403-1368-4814-9a54-7ba842a225f0","Type":"ContainerDied","Data":"cb0967b403986cbb484226a2b3a3fa006df7f2ad5e24abfebe6a924a170b6c52"}
Jul 19 17:24:56.651260 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.651205    2584 scope.go:117] "RemoveContainer" containerID="c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97"
Jul 19 17:24:56.660118 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.660096    2584 scope.go:117] "RemoveContainer" containerID="c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97"
Jul 19 17:24:56.660432 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:24:56.660408    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97\": container with ID starting with c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97 not found: ID does not exist" containerID="c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97"
Jul 19 17:24:56.660547 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.660443    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97"} err="failed to get container status \"c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97\": rpc error: code = NotFound desc = could not find container \"c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97\": container with ID starting with c53655638a2ee92545f0115a8f3d10e10203ccfb69036658abdd7bc2cfcf5b97 not found: ID does not exist"
Jul 19 17:24:56.666799 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.666773    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"]
Jul 19 17:24:56.670025 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:56.669998    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/auth-test-model-8d2863-kserve-57897cb4c-vw4gf"]
Jul 19 17:24:57.143430 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.143388    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr" podUID="52f36157-1532-47e0-94a1-e884f0986386" containerName="main" containerID="cri-o://d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252" gracePeriod=45
Jul 19 17:24:57.410495 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.410463    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:24:57.510783 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.510751    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-home\") pod \"52f36157-1532-47e0-94a1-e884f0986386\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") "
Jul 19 17:24:57.510978 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.510942    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-tmp-dir\") pod \"52f36157-1532-47e0-94a1-e884f0986386\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") "
Jul 19 17:24:57.511047 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511004    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj88w\" (UniqueName: \"kubernetes.io/projected/52f36157-1532-47e0-94a1-e884f0986386-kube-api-access-fj88w\") pod \"52f36157-1532-47e0-94a1-e884f0986386\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") "
Jul 19 17:24:57.511094 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511045    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-home" (OuterVolumeSpecName: "home") pod "52f36157-1532-47e0-94a1-e884f0986386" (UID: "52f36157-1532-47e0-94a1-e884f0986386"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:57.511094 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511063    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f36157-1532-47e0-94a1-e884f0986386-tls-certs\") pod \"52f36157-1532-47e0-94a1-e884f0986386\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") "
Jul 19 17:24:57.511178 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511131    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-dshm\") pod \"52f36157-1532-47e0-94a1-e884f0986386\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") "
Jul 19 17:24:57.511178 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511164    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-model-cache\") pod \"52f36157-1532-47e0-94a1-e884f0986386\" (UID: \"52f36157-1532-47e0-94a1-e884f0986386\") "
Jul 19 17:24:57.511499 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511410    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52f36157-1532-47e0-94a1-e884f0986386" (UID: "52f36157-1532-47e0-94a1-e884f0986386"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:57.511617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511499    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-model-cache" (OuterVolumeSpecName: "model-cache") pod "52f36157-1532-47e0-94a1-e884f0986386" (UID: "52f36157-1532-47e0-94a1-e884f0986386"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:57.511617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511564    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:57.511617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.511584    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:57.513416 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.513393    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f36157-1532-47e0-94a1-e884f0986386-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52f36157-1532-47e0-94a1-e884f0986386" (UID: "52f36157-1532-47e0-94a1-e884f0986386"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:24:57.513512 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.513414    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f36157-1532-47e0-94a1-e884f0986386-kube-api-access-fj88w" (OuterVolumeSpecName: "kube-api-access-fj88w") pod "52f36157-1532-47e0-94a1-e884f0986386" (UID: "52f36157-1532-47e0-94a1-e884f0986386"). InnerVolumeSpecName "kube-api-access-fj88w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:24:57.513697 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.513678    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-dshm" (OuterVolumeSpecName: "dshm") pod "52f36157-1532-47e0-94a1-e884f0986386" (UID: "52f36157-1532-47e0-94a1-e884f0986386"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:24:57.612590 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.612551    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj88w\" (UniqueName: \"kubernetes.io/projected/52f36157-1532-47e0-94a1-e884f0986386-kube-api-access-fj88w\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:57.612590 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.612583    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52f36157-1532-47e0-94a1-e884f0986386-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:57.612590 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.612593    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:57.612590 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.612602    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52f36157-1532-47e0-94a1-e884f0986386-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:24:57.657654 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.657619    2584 generic.go:356] "Generic (PLEG): container finished" podID="52f36157-1532-47e0-94a1-e884f0986386" containerID="d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252" exitCode=0
Jul 19 17:24:57.657862 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.657663    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"
Jul 19 17:24:57.657862 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.657715    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr" event={"ID":"52f36157-1532-47e0-94a1-e884f0986386","Type":"ContainerDied","Data":"d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252"}
Jul 19 17:24:57.657862 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.657761    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr" event={"ID":"52f36157-1532-47e0-94a1-e884f0986386","Type":"ContainerDied","Data":"7ebd7699534d6ba6901bd63b7a01767119f6a033573c1658afe41523ce9625ec"}
Jul 19 17:24:57.657862 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.657785    2584 scope.go:117] "RemoveContainer" containerID="d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252"
Jul 19 17:24:57.667253 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.667234    2584 scope.go:117] "RemoveContainer" containerID="d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252"
Jul 19 17:24:57.667546 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:24:57.667521    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252\": container with ID starting with d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252 not found: ID does not exist" containerID="d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252"
Jul 19 17:24:57.667645 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.667553    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252"} err="failed to get container status \"d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252\": rpc error: code = NotFound desc = could not find container \"d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252\": container with ID starting with d86cbbbbb3c94a542b25fbb86cba7118bf7c298e8e5e0df6cfe424a991177252 not found: ID does not exist"
Jul 19 17:24:57.685839 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.685793    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"]
Jul 19 17:24:57.689121 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:57.689096    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/auth-test-model-0fa4bd-kserve-6dcfbddc47-68xhr"]
Jul 19 17:24:58.158757 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:58.158722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:24:58.206151 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:58.206120    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:24:58.632800 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:58.632718    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52f36157-1532-47e0-94a1-e884f0986386" path="/var/lib/kubelet/pods/52f36157-1532-47e0-94a1-e884f0986386/volumes"
Jul 19 17:24:58.633114 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:24:58.633100    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5f7403-1368-4814-9a54-7ba842a225f0" path="/var/lib/kubelet/pods/9b5f7403-1368-4814-9a54-7ba842a225f0/volumes"
Jul 19 17:25:05.055363 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:05.055326    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:25:05.055777 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:05.055388    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:25:05.068536 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:05.068507    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:25:05.719020 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:05.718990    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:25:13.841017 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.840973    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"]
Jul 19 17:25:13.841457 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.841417    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f36157-1532-47e0-94a1-e884f0986386" containerName="main"
Jul 19 17:25:13.841457 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.841430    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f36157-1532-47e0-94a1-e884f0986386" containerName="main"
Jul 19 17:25:13.841457 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.841439    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5f7403-1368-4814-9a54-7ba842a225f0" containerName="main"
Jul 19 17:25:13.841457 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.841444    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5f7403-1368-4814-9a54-7ba842a225f0" containerName="main"
Jul 19 17:25:13.841596 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.841500    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5f7403-1368-4814-9a54-7ba842a225f0" containerName="main"
Jul 19 17:25:13.841596 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.841509    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f36157-1532-47e0-94a1-e884f0986386" containerName="main"
Jul 19 17:25:13.846228 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.846204    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:13.849187 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.849162    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"kube-root-ca.crt\""
Jul 19 17:25:13.850583 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.850556    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"openshift-service-ca.crt\""
Jul 19 17:25:13.850857 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.850816    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"default-dockercfg-5z74t\""
Jul 19 17:25:13.851024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.851005    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"sub-test-model-0fa4bd-kserve-self-signed-certs\""
Jul 19 17:25:13.851792 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.851767    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"]
Jul 19 17:25:13.869209 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.869176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:25:13.952548 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.952506    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-home\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:13.952548 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.952547    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2bn\" (UniqueName: \"kubernetes.io/projected/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-kube-api-access-cz2bn\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:13.952772 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.952568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tls-certs\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:13.952772 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.952627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-model-cache\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:13.952772 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.952649    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-dshm\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:13.952772 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:13.952679    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tmp-dir\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.053729 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.053690    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-model-cache\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.053729 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.053732    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-dshm\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.053986 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.053759    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tmp-dir\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.053986 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.053831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-home\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.053986 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.053848    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2bn\" (UniqueName: \"kubernetes.io/projected/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-kube-api-access-cz2bn\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.053986 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.053942    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tls-certs\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.054179 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.054161    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-model-cache\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.054239 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.054183    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tmp-dir\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.054332 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.054311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-home\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.056248 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.056216    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-dshm\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.056725 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.056703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tls-certs\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.062072 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.062046    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2bn\" (UniqueName: \"kubernetes.io/projected/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-kube-api-access-cz2bn\") pod \"sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") " pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.157820 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.157783    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:14.304543 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.304510    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"]
Jul 19 17:25:14.308087 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:25:14.308055    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cfbdf46_cc4e_4fc3_82c0_7b9c31b6dcc9.slice/crio-27beaff3003adf06de96727af76c1cc47fe3c7fa1c11d1a8b605fdfffadb7454 WatchSource:0}: Error finding container 27beaff3003adf06de96727af76c1cc47fe3c7fa1c11d1a8b605fdfffadb7454: Status 404 returned error can't find the container with id 27beaff3003adf06de96727af76c1cc47fe3c7fa1c11d1a8b605fdfffadb7454
Jul 19 17:25:14.753912 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.753875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w" event={"ID":"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9","Type":"ContainerStarted","Data":"f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486"}
Jul 19 17:25:14.754093 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.753925    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w" event={"ID":"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9","Type":"ContainerStarted","Data":"27beaff3003adf06de96727af76c1cc47fe3c7fa1c11d1a8b605fdfffadb7454"}
Jul 19 17:25:14.770760 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:14.770706    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w" podStartSLOduration=1.77068594 podStartE2EDuration="1.77068594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:25:14.769682356 +0000 UTC m=+2824.674933781" watchObservedRunningTime="2026-07-19 17:25:14.77068594 +0000 UTC m=+2824.675937364"
Jul 19 17:25:24.158624 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:24.158588    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:24.159035 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:24.158675    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:24.170923 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:24.170895    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:25:24.833756 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:25:24.833728    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:26:01.647075 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:01.646986    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"]
Jul 19 17:26:01.680718 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:01.680676    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:26:02.574259 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:02.574226    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"]
Jul 19 17:26:02.751909 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:02.751874    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:26:11.035811 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.035775    2584 scope.go:117] "RemoveContainer" containerID="6f268541986e1896dcffd0cf2878904ad883a2683dafcaf838d4993854044ad8"
Jul 19 17:26:11.045522 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.045496    2584 scope.go:117] "RemoveContainer" containerID="767c92ef47364e789529974b5626589a9210684553c1841288ac09ca23def92e"
Jul 19 17:26:11.451067 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.451033    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"]
Jul 19 17:26:11.456675 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.456653    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.459224 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.459194    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8d2863\"/\"rate-test-model-8d2863-kserve-self-signed-certs\""
Jul 19 17:26:11.471003 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.470969    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"]
Jul 19 17:26:11.479869 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.479834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:26:11.584617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.584580    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tls-certs\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.584617 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.584618    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-dshm\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.584876 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.584643    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tmp-dir\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.584876 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.584761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-home\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.584876 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.584792    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfgs\" (UniqueName: \"kubernetes.io/projected/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-kube-api-access-csfgs\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.585024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.584906    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-model-cache\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.685845 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.685811    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tls-certs\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.685845 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.685845    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-dshm\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686104 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.685868    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tmp-dir\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686104 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.686000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-home\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686104 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.686034    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfgs\" (UniqueName: \"kubernetes.io/projected/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-kube-api-access-csfgs\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686266 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.686119    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-model-cache\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686266 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.686233    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tmp-dir\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686409 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.686391    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-home\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.686513 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.686491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-model-cache\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.688521 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.688491    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-dshm\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.688815 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.688797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tls-certs\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.695193 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.695154    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfgs\" (UniqueName: \"kubernetes.io/projected/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-kube-api-access-csfgs\") pod \"rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") " pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.765741 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.765649    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:11.914978 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:11.914949    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"]
Jul 19 17:26:11.916934 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:26:11.916911    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc73cece_ca63_4f6b_a34b_9ec93a91e48f.slice/crio-f611d00b67548689d311de8100ff43dc3a57698e9a3eb0352a9570bec1cc7b06 WatchSource:0}: Error finding container f611d00b67548689d311de8100ff43dc3a57698e9a3eb0352a9570bec1cc7b06: Status 404 returned error can't find the container with id f611d00b67548689d311de8100ff43dc3a57698e9a3eb0352a9570bec1cc7b06
Jul 19 17:26:12.086205 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:12.086113    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr" event={"ID":"bc73cece-ca63-4f6b-a34b-9ec93a91e48f","Type":"ContainerStarted","Data":"6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e"}
Jul 19 17:26:12.086205 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:12.086161    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr" event={"ID":"bc73cece-ca63-4f6b-a34b-9ec93a91e48f","Type":"ContainerStarted","Data":"f611d00b67548689d311de8100ff43dc3a57698e9a3eb0352a9570bec1cc7b06"}
Jul 19 17:26:12.106174 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:12.106123    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr" podStartSLOduration=1.106104018 podStartE2EDuration="1.106104018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:26:12.104245785 +0000 UTC m=+2882.009497210" watchObservedRunningTime="2026-07-19 17:26:12.106104018 +0000 UTC m=+2882.011355486"
Jul 19 17:26:16.667471 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:16.667426    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq" podUID="958303f3-e33a-4904-919f-6973276c43af" containerName="main" containerID="cri-o://01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73" gracePeriod=45
Jul 19 17:26:16.935059 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:16.935031    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:26:17.039655 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039615    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-model-cache\") pod \"958303f3-e33a-4904-919f-6973276c43af\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") "
Jul 19 17:26:17.039853 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039692    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-tmp-dir\") pod \"958303f3-e33a-4904-919f-6973276c43af\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") "
Jul 19 17:26:17.039853 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039748    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhrfh\" (UniqueName: \"kubernetes.io/projected/958303f3-e33a-4904-919f-6973276c43af-kube-api-access-nhrfh\") pod \"958303f3-e33a-4904-919f-6973276c43af\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") "
Jul 19 17:26:17.039853 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039781    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-home\") pod \"958303f3-e33a-4904-919f-6973276c43af\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") "
Jul 19 17:26:17.039853 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039807    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-dshm\") pod \"958303f3-e33a-4904-919f-6973276c43af\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") "
Jul 19 17:26:17.040086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039873    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958303f3-e33a-4904-919f-6973276c43af-tls-certs\") pod \"958303f3-e33a-4904-919f-6973276c43af\" (UID: \"958303f3-e33a-4904-919f-6973276c43af\") "
Jul 19 17:26:17.040086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-model-cache" (OuterVolumeSpecName: "model-cache") pod "958303f3-e33a-4904-919f-6973276c43af" (UID: "958303f3-e33a-4904-919f-6973276c43af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.040086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.039983    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "958303f3-e33a-4904-919f-6973276c43af" (UID: "958303f3-e33a-4904-919f-6973276c43af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.040086 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.040074    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-home" (OuterVolumeSpecName: "home") pod "958303f3-e33a-4904-919f-6973276c43af" (UID: "958303f3-e33a-4904-919f-6973276c43af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.040257 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.040127    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.040257 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.040146    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.042078 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.042052    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-dshm" (OuterVolumeSpecName: "dshm") pod "958303f3-e33a-4904-919f-6973276c43af" (UID: "958303f3-e33a-4904-919f-6973276c43af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.042078 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.042066    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/958303f3-e33a-4904-919f-6973276c43af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "958303f3-e33a-4904-919f-6973276c43af" (UID: "958303f3-e33a-4904-919f-6973276c43af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:26:17.042078 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.042067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958303f3-e33a-4904-919f-6973276c43af-kube-api-access-nhrfh" (OuterVolumeSpecName: "kube-api-access-nhrfh") pod "958303f3-e33a-4904-919f-6973276c43af" (UID: "958303f3-e33a-4904-919f-6973276c43af"). InnerVolumeSpecName "kube-api-access-nhrfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:26:17.111779 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.111746    2584 generic.go:356] "Generic (PLEG): container finished" podID="958303f3-e33a-4904-919f-6973276c43af" containerID="01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73" exitCode=0
Jul 19 17:26:17.111961 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.111793    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"
Jul 19 17:26:17.111961 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.111826    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq" event={"ID":"958303f3-e33a-4904-919f-6973276c43af","Type":"ContainerDied","Data":"01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73"}
Jul 19 17:26:17.111961 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.111867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq" event={"ID":"958303f3-e33a-4904-919f-6973276c43af","Type":"ContainerDied","Data":"c4d262da76fe56ffb9f21858e708ce321a72662293ac6b5ec3ed96f53134fc20"}
Jul 19 17:26:17.111961 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.111883    2584 scope.go:117] "RemoveContainer" containerID="01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73"
Jul 19 17:26:17.121818 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.121801    2584 scope.go:117] "RemoveContainer" containerID="01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73"
Jul 19 17:26:17.122080 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:26:17.122062    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73\": container with ID starting with 01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73 not found: ID does not exist" containerID="01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73"
Jul 19 17:26:17.122151 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.122093    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73"} err="failed to get container status \"01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73\": rpc error: code = NotFound desc = could not find container \"01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73\": container with ID starting with 01ae112cf8d91093b844183bcf83b6fc69599dc15b7aedd4aee1ce00968c2f73 not found: ID does not exist"
Jul 19 17:26:17.133214 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.133189    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"]
Jul 19 17:26:17.137651 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.137629    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/sub-test-model-8d2863-kserve-d9dc7556f-m9mlq"]
Jul 19 17:26:17.140920 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.140902    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhrfh\" (UniqueName: \"kubernetes.io/projected/958303f3-e33a-4904-919f-6973276c43af-kube-api-access-nhrfh\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.140969 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.140923    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.140969 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.140937    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958303f3-e33a-4904-919f-6973276c43af-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.140969 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.140948    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958303f3-e33a-4904-919f-6973276c43af-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.594922 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.594850    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w" podUID="1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" containerName="main" containerID="cri-o://f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486" gracePeriod=45
Jul 19 17:26:17.874055 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.874028    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:26:17.947469 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947435    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-dshm\") pod \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") "
Jul 19 17:26:17.947639 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947487    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tls-certs\") pod \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") "
Jul 19 17:26:17.947639 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947547    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tmp-dir\") pod \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") "
Jul 19 17:26:17.947639 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947615    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-home\") pod \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") "
Jul 19 17:26:17.947768 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947644    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-model-cache\") pod \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") "
Jul 19 17:26:17.947768 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947673    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz2bn\" (UniqueName: \"kubernetes.io/projected/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-kube-api-access-cz2bn\") pod \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\" (UID: \"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9\") "
Jul 19 17:26:17.947868 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947822    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" (UID: "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.947926 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947900    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-home" (OuterVolumeSpecName: "home") pod "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" (UID: "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.947985 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.947938    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" (UID: "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:17.948100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.948081    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.948154 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.948102    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.948154 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.948111    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:17.949843 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.949820    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" (UID: "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:26:17.949916 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.949855    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-kube-api-access-cz2bn" (OuterVolumeSpecName: "kube-api-access-cz2bn") pod "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" (UID: "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9"). InnerVolumeSpecName "kube-api-access-cz2bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:26:17.949916 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:17.949879    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-dshm" (OuterVolumeSpecName: "dshm") pod "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" (UID: "1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:26:18.049264 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.049222    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2bn\" (UniqueName: \"kubernetes.io/projected/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-kube-api-access-cz2bn\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:18.049264 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.049256    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:18.049264 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.049266    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:26:18.118118 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.118030    2584 generic.go:356] "Generic (PLEG): container finished" podID="1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" containerID="f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486" exitCode=0
Jul 19 17:26:18.118118 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.118077    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"
Jul 19 17:26:18.118401 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.118121    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w" event={"ID":"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9","Type":"ContainerDied","Data":"f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486"}
Jul 19 17:26:18.118401 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.118169    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w" event={"ID":"1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9","Type":"ContainerDied","Data":"27beaff3003adf06de96727af76c1cc47fe3c7fa1c11d1a8b605fdfffadb7454"}
Jul 19 17:26:18.118401 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.118193    2584 scope.go:117] "RemoveContainer" containerID="f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486"
Jul 19 17:26:18.131449 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.131427    2584 scope.go:117] "RemoveContainer" containerID="f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486"
Jul 19 17:26:18.131754 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:26:18.131732    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486\": container with ID starting with f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486 not found: ID does not exist" containerID="f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486"
Jul 19 17:26:18.131800 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.131765    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486"} err="failed to get container status \"f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486\": rpc error: code = NotFound desc = could not find container \"f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486\": container with ID starting with f2763965303aaa7682ce2298f44948af71e3bdc8f1cccd2f8c16aa2661c2e486 not found: ID does not exist"
Jul 19 17:26:18.139931 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.139904    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"]
Jul 19 17:26:18.143332 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.143311    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/sub-test-model-0fa4bd-kserve-68ff7884bf-xcc8w"]
Jul 19 17:26:18.633142 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.633113    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" path="/var/lib/kubelet/pods/1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9/volumes"
Jul 19 17:26:18.633546 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:18.633532    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958303f3-e33a-4904-919f-6973276c43af" path="/var/lib/kubelet/pods/958303f3-e33a-4904-919f-6973276c43af/volumes"
Jul 19 17:26:21.766837 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:21.766801    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:21.766837 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:21.766844    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:21.780565 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:21.780535    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:22.151759 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:22.151736    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:26:39.475088 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475049    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"]
Jul 19 17:26:39.475630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475460    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958303f3-e33a-4904-919f-6973276c43af" containerName="main"
Jul 19 17:26:39.475630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475474    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="958303f3-e33a-4904-919f-6973276c43af" containerName="main"
Jul 19 17:26:39.475630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475501    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" containerName="main"
Jul 19 17:26:39.475630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475508    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" containerName="main"
Jul 19 17:26:39.475630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475568    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cfbdf46-cc4e-4fc3-82c0-7b9c31b6dcc9" containerName="main"
Jul 19 17:26:39.475630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.475576    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="958303f3-e33a-4904-919f-6973276c43af" containerName="main"
Jul 19 17:26:39.478832 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.478812    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.482414 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.482385    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"openshift-service-ca.crt\""
Jul 19 17:26:39.482414 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.482393    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"kube-root-ca.crt\""
Jul 19 17:26:39.482834 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.482817    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"default-dockercfg-5z74t\""
Jul 19 17:26:39.483252 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.483239    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0fa4bd\"/\"rate-test-model-0fa4bd-kserve-self-signed-certs\""
Jul 19 17:26:39.490329 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.490304    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"]
Jul 19 17:26:39.555925 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.555890    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57faf389-b0c2-4377-87c4-ffdf84d09767-tls-certs\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.556097 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.555936    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-tmp-dir\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.556097 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.556035    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-model-cache\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.556097 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.556065    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-home\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.556097 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.556088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjw8c\" (UniqueName: \"kubernetes.io/projected/57faf389-b0c2-4377-87c4-ffdf84d09767-kube-api-access-sjw8c\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.556237 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.556166    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-dshm\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.583504 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.583468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:26:39.657346 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-dshm\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.657578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57faf389-b0c2-4377-87c4-ffdf84d09767-tls-certs\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.657578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-tmp-dir\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.657578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657517    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-model-cache\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.657578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-home\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.657578 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjw8c\" (UniqueName: \"kubernetes.io/projected/57faf389-b0c2-4377-87c4-ffdf84d09767-kube-api-access-sjw8c\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.657933 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657904    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-tmp-dir\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.658034 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.657915    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-home\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.658034 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.658011    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-model-cache\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.659884 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.659862    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-dshm\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.660051 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.660035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57faf389-b0c2-4377-87c4-ffdf84d09767-tls-certs\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.676254 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.676229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjw8c\" (UniqueName: \"kubernetes.io/projected/57faf389-b0c2-4377-87c4-ffdf84d09767-kube-api-access-sjw8c\") pod \"rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") " pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.788512 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.788412    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:39.936519 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:39.936495    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"]
Jul 19 17:26:39.938802 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:26:39.938774    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57faf389_b0c2_4377_87c4_ffdf84d09767.slice/crio-faf82592f8eed4220f76a82ae05317486cdec07effbf00e1e6ccede844142a99 WatchSource:0}: Error finding container faf82592f8eed4220f76a82ae05317486cdec07effbf00e1e6ccede844142a99: Status 404 returned error can't find the container with id faf82592f8eed4220f76a82ae05317486cdec07effbf00e1e6ccede844142a99
Jul 19 17:26:40.231270 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:40.231223    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7" event={"ID":"57faf389-b0c2-4377-87c4-ffdf84d09767","Type":"ContainerStarted","Data":"4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5"}
Jul 19 17:26:40.231270 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:40.231290    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7" event={"ID":"57faf389-b0c2-4377-87c4-ffdf84d09767","Type":"ContainerStarted","Data":"faf82592f8eed4220f76a82ae05317486cdec07effbf00e1e6ccede844142a99"}
Jul 19 17:26:40.264256 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:40.264206    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7" podStartSLOduration=1.264183488 podStartE2EDuration="1.264183488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 17:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:26:40.263409989 +0000 UTC m=+2910.168661404" watchObservedRunningTime="2026-07-19 17:26:40.264183488 +0000 UTC m=+2910.169434914"
Jul 19 17:26:49.788946 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:49.788911    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:49.789425 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:49.788963    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:49.803135 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:49.803098    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:50.299352 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:50.299324    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:26:59.815129 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:26:59.815093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:02.424100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:02.424066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:02.525426 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:02.525398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:11.069121 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:11.069086    2584 scope.go:117] "RemoveContainer" containerID="321bcdf40afa60e90f66e4459acc9c27670e37220b877acdbbdb14813d04b5f1"
Jul 19 17:27:11.077787 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:11.077764    2584 scope.go:117] "RemoveContainer" containerID="83fd6aa04f2ab821f7b2a4618fe95366cabbef51eb42902c6a1f089ddc5e90b4"
Jul 19 17:27:11.421780 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:11.421743    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:11.523078 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:11.523046    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:22.827247 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:22.827209    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"]
Jul 19 17:27:22.845514 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:22.845476    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:23.930889 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:23.930851    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"]
Jul 19 17:27:23.954497 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:23.954455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:28.213196 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:28.213157    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:28.292616 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:28.292582    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:37.849584 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:37.849502    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr" podUID="bc73cece-ca63-4f6b-a34b-9ec93a91e48f" containerName="main" containerID="cri-o://6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e" gracePeriod=45
Jul 19 17:27:38.107175 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.107106    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:27:38.218588 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218551    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-model-cache\") pod \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") "
Jul 19 17:27:38.218588 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218591    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-dshm\") pod \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") "
Jul 19 17:27:38.218807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218641    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tmp-dir\") pod \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") "
Jul 19 17:27:38.218807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218677    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-home\") pod \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") "
Jul 19 17:27:38.218807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218720    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csfgs\" (UniqueName: \"kubernetes.io/projected/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-kube-api-access-csfgs\") pod \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") "
Jul 19 17:27:38.218807 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218766    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tls-certs\") pod \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\" (UID: \"bc73cece-ca63-4f6b-a34b-9ec93a91e48f\") "
Jul 19 17:27:38.218985 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218880    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc73cece-ca63-4f6b-a34b-9ec93a91e48f" (UID: "bc73cece-ca63-4f6b-a34b-9ec93a91e48f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:38.218985 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.218951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc73cece-ca63-4f6b-a34b-9ec93a91e48f" (UID: "bc73cece-ca63-4f6b-a34b-9ec93a91e48f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:38.219128 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.219112    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:38.219207 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.219133    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:38.219295 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.219260    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-home" (OuterVolumeSpecName: "home") pod "bc73cece-ca63-4f6b-a34b-9ec93a91e48f" (UID: "bc73cece-ca63-4f6b-a34b-9ec93a91e48f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:38.221094 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.221067    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc73cece-ca63-4f6b-a34b-9ec93a91e48f" (UID: "bc73cece-ca63-4f6b-a34b-9ec93a91e48f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:27:38.221209 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.221152    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-dshm" (OuterVolumeSpecName: "dshm") pod "bc73cece-ca63-4f6b-a34b-9ec93a91e48f" (UID: "bc73cece-ca63-4f6b-a34b-9ec93a91e48f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:38.221209 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.221172    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-kube-api-access-csfgs" (OuterVolumeSpecName: "kube-api-access-csfgs") pod "bc73cece-ca63-4f6b-a34b-9ec93a91e48f" (UID: "bc73cece-ca63-4f6b-a34b-9ec93a91e48f"). InnerVolumeSpecName "kube-api-access-csfgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:27:38.320273 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.320240    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:38.320273 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.320267    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfgs\" (UniqueName: \"kubernetes.io/projected/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-kube-api-access-csfgs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:38.320492 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.320305    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:38.320492 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.320314    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc73cece-ca63-4f6b-a34b-9ec93a91e48f-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:38.532540 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.532502    2584 generic.go:356] "Generic (PLEG): container finished" podID="bc73cece-ca63-4f6b-a34b-9ec93a91e48f" containerID="6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e" exitCode=0
Jul 19 17:27:38.532743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.532556    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"
Jul 19 17:27:38.532743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.532583    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr" event={"ID":"bc73cece-ca63-4f6b-a34b-9ec93a91e48f","Type":"ContainerDied","Data":"6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e"}
Jul 19 17:27:38.532743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.532620    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr" event={"ID":"bc73cece-ca63-4f6b-a34b-9ec93a91e48f","Type":"ContainerDied","Data":"f611d00b67548689d311de8100ff43dc3a57698e9a3eb0352a9570bec1cc7b06"}
Jul 19 17:27:38.532743 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.532639    2584 scope.go:117] "RemoveContainer" containerID="6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e"
Jul 19 17:27:38.542258 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.542243    2584 scope.go:117] "RemoveContainer" containerID="6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e"
Jul 19 17:27:38.542611 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:27:38.542592    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e\": container with ID starting with 6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e not found: ID does not exist" containerID="6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e"
Jul 19 17:27:38.542679 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.542623    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e"} err="failed to get container status \"6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e\": rpc error: code = NotFound desc = could not find container \"6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e\": container with ID starting with 6cf60ca5db5dd476d40f64bdf33dd39de638674d49911a7fc5d885c3eec3909e not found: ID does not exist"
Jul 19 17:27:38.553708 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.553678    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"]
Jul 19 17:27:38.555660 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.555639    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8d2863/rate-test-model-8d2863-kserve-7cf5785d8c-bjxcr"]
Jul 19 17:27:38.633991 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.633953    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc73cece-ca63-4f6b-a34b-9ec93a91e48f" path="/var/lib/kubelet/pods/bc73cece-ca63-4f6b-a34b-9ec93a91e48f/volumes"
Jul 19 17:27:38.951569 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:38.951529    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7" podUID="57faf389-b0c2-4377-87c4-ffdf84d09767" containerName="main" containerID="cri-o://4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5" gracePeriod=45
Jul 19 17:27:39.209516 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.209457    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:27:39.329550 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329512    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-model-cache\") pod \"57faf389-b0c2-4377-87c4-ffdf84d09767\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") "
Jul 19 17:27:39.329714 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329578    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjw8c\" (UniqueName: \"kubernetes.io/projected/57faf389-b0c2-4377-87c4-ffdf84d09767-kube-api-access-sjw8c\") pod \"57faf389-b0c2-4377-87c4-ffdf84d09767\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") "
Jul 19 17:27:39.329714 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329635    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-dshm\") pod \"57faf389-b0c2-4377-87c4-ffdf84d09767\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") "
Jul 19 17:27:39.329714 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329653    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57faf389-b0c2-4377-87c4-ffdf84d09767-tls-certs\") pod \"57faf389-b0c2-4377-87c4-ffdf84d09767\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") "
Jul 19 17:27:39.329714 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329674    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-home\") pod \"57faf389-b0c2-4377-87c4-ffdf84d09767\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") "
Jul 19 17:27:39.329933 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329749    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-tmp-dir\") pod \"57faf389-b0c2-4377-87c4-ffdf84d09767\" (UID: \"57faf389-b0c2-4377-87c4-ffdf84d09767\") "
Jul 19 17:27:39.329933 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329848    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-model-cache" (OuterVolumeSpecName: "model-cache") pod "57faf389-b0c2-4377-87c4-ffdf84d09767" (UID: "57faf389-b0c2-4377-87c4-ffdf84d09767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:39.330042 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.329963    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-home" (OuterVolumeSpecName: "home") pod "57faf389-b0c2-4377-87c4-ffdf84d09767" (UID: "57faf389-b0c2-4377-87c4-ffdf84d09767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:39.330042 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.330016    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57faf389-b0c2-4377-87c4-ffdf84d09767" (UID: "57faf389-b0c2-4377-87c4-ffdf84d09767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:39.330174 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.330102    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-model-cache\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:39.330174 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.330123    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-home\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:39.331957 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.331933    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57faf389-b0c2-4377-87c4-ffdf84d09767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57faf389-b0c2-4377-87c4-ffdf84d09767" (UID: "57faf389-b0c2-4377-87c4-ffdf84d09767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:27:39.332057 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.331990    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57faf389-b0c2-4377-87c4-ffdf84d09767-kube-api-access-sjw8c" (OuterVolumeSpecName: "kube-api-access-sjw8c") pod "57faf389-b0c2-4377-87c4-ffdf84d09767" (UID: "57faf389-b0c2-4377-87c4-ffdf84d09767"). InnerVolumeSpecName "kube-api-access-sjw8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:27:39.332057 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.332019    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-dshm" (OuterVolumeSpecName: "dshm") pod "57faf389-b0c2-4377-87c4-ffdf84d09767" (UID: "57faf389-b0c2-4377-87c4-ffdf84d09767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 17:27:39.431632 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.431595    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-tmp-dir\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:39.431632 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.431629    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjw8c\" (UniqueName: \"kubernetes.io/projected/57faf389-b0c2-4377-87c4-ffdf84d09767-kube-api-access-sjw8c\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:39.431632 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.431639    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57faf389-b0c2-4377-87c4-ffdf84d09767-dshm\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:39.431854 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.431647    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57faf389-b0c2-4377-87c4-ffdf84d09767-tls-certs\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:39.539270 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.539158    2584 generic.go:356] "Generic (PLEG): container finished" podID="57faf389-b0c2-4377-87c4-ffdf84d09767" containerID="4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5" exitCode=0
Jul 19 17:27:39.539270 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.539210    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"
Jul 19 17:27:39.539270 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.539243    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7" event={"ID":"57faf389-b0c2-4377-87c4-ffdf84d09767","Type":"ContainerDied","Data":"4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5"}
Jul 19 17:27:39.539593 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.539310    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7" event={"ID":"57faf389-b0c2-4377-87c4-ffdf84d09767","Type":"ContainerDied","Data":"faf82592f8eed4220f76a82ae05317486cdec07effbf00e1e6ccede844142a99"}
Jul 19 17:27:39.539593 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.539335    2584 scope.go:117] "RemoveContainer" containerID="4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5"
Jul 19 17:27:39.549296 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.549253    2584 scope.go:117] "RemoveContainer" containerID="4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5"
Jul 19 17:27:39.549626 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:27:39.549606    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5\": container with ID starting with 4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5 not found: ID does not exist" containerID="4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5"
Jul 19 17:27:39.549707 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.549637    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5"} err="failed to get container status \"4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5\": rpc error: code = NotFound desc = could not find container \"4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5\": container with ID starting with 4c7237455dce8318e9c3eb726b05dea7d5c39640a6ed25d8d394bea79c6858a5 not found: ID does not exist"
Jul 19 17:27:39.562986 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.562959    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"]
Jul 19 17:27:39.567999 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.567973    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0fa4bd/rate-test-model-0fa4bd-kserve-89bcf5f55-xl2t7"]
Jul 19 17:27:39.940992 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.940951    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7448-139963577788576"]
Jul 19 17:27:39.941368 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.941356    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc73cece-ca63-4f6b-a34b-9ec93a91e48f" containerName="main"
Jul 19 17:27:39.941429 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.941370    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc73cece-ca63-4f6b-a34b-9ec93a91e48f" containerName="main"
Jul 19 17:27:39.941429 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.941379    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57faf389-b0c2-4377-87c4-ffdf84d09767" containerName="main"
Jul 19 17:27:39.941429 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.941384    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="57faf389-b0c2-4377-87c4-ffdf84d09767" containerName="main"
Jul 19 17:27:39.941534 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.941437    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc73cece-ca63-4f6b-a34b-9ec93a91e48f" containerName="main"
Jul 19 17:27:39.941534 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.941447    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="57faf389-b0c2-4377-87c4-ffdf84d09767" containerName="main"
Jul 19 17:27:39.946034 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.946005    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:39.948440 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.948404    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7448-139963577788576"]
Jul 19 17:27:39.948555 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:39.948525    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cm2bs\""
Jul 19 17:27:40.035768 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.035729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78psb\" (UniqueName: \"kubernetes.io/projected/d1bfecec-337e-4e66-93ad-d9899a688a7b-kube-api-access-78psb\") pod \"test-curl-7448-139963577788576\" (UID: \"d1bfecec-337e-4e66-93ad-d9899a688a7b\") " pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:40.137209 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.137171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78psb\" (UniqueName: \"kubernetes.io/projected/d1bfecec-337e-4e66-93ad-d9899a688a7b-kube-api-access-78psb\") pod \"test-curl-7448-139963577788576\" (UID: \"d1bfecec-337e-4e66-93ad-d9899a688a7b\") " pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:40.145563 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.145527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78psb\" (UniqueName: \"kubernetes.io/projected/d1bfecec-337e-4e66-93ad-d9899a688a7b-kube-api-access-78psb\") pod \"test-curl-7448-139963577788576\" (UID: \"d1bfecec-337e-4e66-93ad-d9899a688a7b\") " pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:40.256501 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.256408    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:40.391508 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.391482    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7448-139963577788576"]
Jul 19 17:27:40.393354 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:27:40.393326    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1bfecec_337e_4e66_93ad_d9899a688a7b.slice/crio-dd379e590724fcb2b81105ba0a6ba45553c54162882beb0fdc854d74fb365551 WatchSource:0}: Error finding container dd379e590724fcb2b81105ba0a6ba45553c54162882beb0fdc854d74fb365551: Status 404 returned error can't find the container with id dd379e590724fcb2b81105ba0a6ba45553c54162882beb0fdc854d74fb365551
Jul 19 17:27:40.394582 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.394561    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 17:27:40.549295 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.549180    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7448-139963577788576" event={"ID":"d1bfecec-337e-4e66-93ad-d9899a688a7b","Type":"ContainerStarted","Data":"dd379e590724fcb2b81105ba0a6ba45553c54162882beb0fdc854d74fb365551"}
Jul 19 17:27:40.632406 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:40.632377    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57faf389-b0c2-4377-87c4-ffdf84d09767" path="/var/lib/kubelet/pods/57faf389-b0c2-4377-87c4-ffdf84d09767/volumes"
Jul 19 17:27:41.555006 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:41.554968    2584 generic.go:356] "Generic (PLEG): container finished" podID="d1bfecec-337e-4e66-93ad-d9899a688a7b" containerID="28569e74eaa371ac6087ba051193a8adf7aebae281d79c625c4e8e49e8a8c3c6" exitCode=0
Jul 19 17:27:41.555494 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:41.555047    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7448-139963577788576" event={"ID":"d1bfecec-337e-4e66-93ad-d9899a688a7b","Type":"ContainerDied","Data":"28569e74eaa371ac6087ba051193a8adf7aebae281d79c625c4e8e49e8a8c3c6"}
Jul 19 17:27:42.705175 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.705148    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:42.731637 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.731610    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7448-139963577788576_d1bfecec-337e-4e66-93ad-d9899a688a7b/test-curl-7448-139963577788576/0.log"
Jul 19 17:27:42.756473 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.756436    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7448-139963577788576"]
Jul 19 17:27:42.759905 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.759874    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7448-139963577788576"]
Jul 19 17:27:42.861540 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.861450    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78psb\" (UniqueName: \"kubernetes.io/projected/d1bfecec-337e-4e66-93ad-d9899a688a7b-kube-api-access-78psb\") pod \"d1bfecec-337e-4e66-93ad-d9899a688a7b\" (UID: \"d1bfecec-337e-4e66-93ad-d9899a688a7b\") "
Jul 19 17:27:42.863786 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.863761    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bfecec-337e-4e66-93ad-d9899a688a7b-kube-api-access-78psb" (OuterVolumeSpecName: "kube-api-access-78psb") pod "d1bfecec-337e-4e66-93ad-d9899a688a7b" (UID: "d1bfecec-337e-4e66-93ad-d9899a688a7b"). InnerVolumeSpecName "kube-api-access-78psb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:27:42.962509 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.962474    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78psb\" (UniqueName: \"kubernetes.io/projected/d1bfecec-337e-4e66-93ad-d9899a688a7b-kube-api-access-78psb\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:42.977682 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.977646    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"]
Jul 19 17:27:42.978058 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.978042    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bfecec-337e-4e66-93ad-d9899a688a7b" containerName="test-curl-7448-139963577788576"
Jul 19 17:27:42.978137 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.978060    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bfecec-337e-4e66-93ad-d9899a688a7b" containerName="test-curl-7448-139963577788576"
Jul 19 17:27:42.978137 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.978134    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bfecec-337e-4e66-93ad-d9899a688a7b" containerName="test-curl-7448-139963577788576"
Jul 19 17:27:42.981223 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.981206    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:42.983718 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.983682    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:27:42.988024 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:42.987994    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"]
Jul 19 17:27:43.063703 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.063671    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423728bb-fae5-4d8b-874b-976f6540ae37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq\" (UID: \"423728bb-fae5-4d8b-874b-976f6540ae37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:43.165346 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.165253    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423728bb-fae5-4d8b-874b-976f6540ae37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq\" (UID: \"423728bb-fae5-4d8b-874b-976f6540ae37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:43.165993 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.165966    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423728bb-fae5-4d8b-874b-976f6540ae37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq\" (UID: \"423728bb-fae5-4d8b-874b-976f6540ae37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:43.295062 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.295025    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:43.439625 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.439599    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"]
Jul 19 17:27:43.441879 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:27:43.441821    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423728bb_fae5_4d8b_874b_976f6540ae37.slice/crio-9238d49b9318cec7c9b270f12b4664f60ca2035f9e64a5d3f3d3b82ffa7beb4d WatchSource:0}: Error finding container 9238d49b9318cec7c9b270f12b4664f60ca2035f9e64a5d3f3d3b82ffa7beb4d: Status 404 returned error can't find the container with id 9238d49b9318cec7c9b270f12b4664f60ca2035f9e64a5d3f3d3b82ffa7beb4d
Jul 19 17:27:43.566264 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.566227    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq" event={"ID":"423728bb-fae5-4d8b-874b-976f6540ae37","Type":"ContainerStarted","Data":"9238d49b9318cec7c9b270f12b4664f60ca2035f9e64a5d3f3d3b82ffa7beb4d"}
Jul 19 17:27:43.567560 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.567539    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:43.567677 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.567591    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd379e590724fcb2b81105ba0a6ba45553c54162882beb0fdc854d74fb365551"
Jul 19 17:27:43.577900 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:27:43.577861    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7448-139963577788576\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="d1bfecec-337e-4e66-93ad-d9899a688a7b" pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:43.589311 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.589270    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:43.690222 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:43.690192    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:44.573682 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:44.573648    2584 generic.go:356] "Generic (PLEG): container finished" podID="423728bb-fae5-4d8b-874b-976f6540ae37" containerID="1faa85ccb51c01c5eab72137875f37f4c211fa96c67daada81956f3ef8833147" exitCode=0
Jul 19 17:27:44.574065 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:44.573742    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq" event={"ID":"423728bb-fae5-4d8b-874b-976f6540ae37","Type":"ContainerDied","Data":"1faa85ccb51c01c5eab72137875f37f4c211fa96c67daada81956f3ef8833147"}
Jul 19 17:27:44.588871 ip-10-0-142-78 kubenswrapper[2584]: E0719 17:27:44.588839    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7448-139963577788576\" is forbidden: User \"system:node:ip-10-0-142-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-78.ec2.internal' and this object" podUID="d1bfecec-337e-4e66-93ad-d9899a688a7b" pod="opendatahub/test-curl-7448-139963577788576"
Jul 19 17:27:44.633161 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:44.633129    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1bfecec-337e-4e66-93ad-d9899a688a7b" path="/var/lib/kubelet/pods/d1bfecec-337e-4e66-93ad-d9899a688a7b/volumes"
Jul 19 17:27:45.719006 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:45.718976    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:45.886424 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:45.886326    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423728bb-fae5-4d8b-874b-976f6540ae37-maas-api-service-ca\") pod \"423728bb-fae5-4d8b-874b-976f6540ae37\" (UID: \"423728bb-fae5-4d8b-874b-976f6540ae37\") "
Jul 19 17:27:45.886778 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:45.886739    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423728bb-fae5-4d8b-874b-976f6540ae37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "423728bb-fae5-4d8b-874b-976f6540ae37" (UID: "423728bb-fae5-4d8b-874b-976f6540ae37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:27:45.993721 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:45.993626    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423728bb-fae5-4d8b-874b-976f6540ae37-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:46.585644 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:46.585609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq" event={"ID":"423728bb-fae5-4d8b-874b-976f6540ae37","Type":"ContainerDied","Data":"9238d49b9318cec7c9b270f12b4664f60ca2035f9e64a5d3f3d3b82ffa7beb4d"}
Jul 19 17:27:46.585644 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:46.585635    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"
Jul 19 17:27:46.585835 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:46.585683    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9238d49b9318cec7c9b270f12b4664f60ca2035f9e64a5d3f3d3b82ffa7beb4d"
Jul 19 17:27:53.366123 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.366084    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"]
Jul 19 17:27:53.367019 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.366981    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" podUID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" containerName="maas-api" containerID="cri-o://9aba380302518ee3c1b22d0ce1507728f04f1bdf9119d963a3dcbe39308fce03" gracePeriod=30
Jul 19 17:27:53.391720 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.389404    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:53.619677 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.619602    2584 generic.go:356] "Generic (PLEG): container finished" podID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" containerID="9aba380302518ee3c1b22d0ce1507728f04f1bdf9119d963a3dcbe39308fce03" exitCode=0
Jul 19 17:27:53.619829 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.619677    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" event={"ID":"3b962c19-0a4c-44bf-8f97-e7525a8f513a","Type":"ContainerDied","Data":"9aba380302518ee3c1b22d0ce1507728f04f1bdf9119d963a3dcbe39308fce03"}
Jul 19 17:27:53.619829 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.619729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" event={"ID":"3b962c19-0a4c-44bf-8f97-e7525a8f513a","Type":"ContainerDied","Data":"f3db33ed7d1fbaa89c01c2283c6607000503790cfb20f99295438ec1a0b010eb"}
Jul 19 17:27:53.619829 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.619745    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3db33ed7d1fbaa89c01c2283c6607000503790cfb20f99295438ec1a0b010eb"
Jul 19 17:27:53.623940 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.623920    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:27:53.761502 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.761464    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls\") pod \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") "
Jul 19 17:27:53.761689 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.761518    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b962c19-0a4c-44bf-8f97-e7525a8f513a-openshift-service-ca\") pod \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") "
Jul 19 17:27:53.761689 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.761571    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l758q\" (UniqueName: \"kubernetes.io/projected/3b962c19-0a4c-44bf-8f97-e7525a8f513a-kube-api-access-l758q\") pod \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\" (UID: \"3b962c19-0a4c-44bf-8f97-e7525a8f513a\") "
Jul 19 17:27:53.761965 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.761932    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b962c19-0a4c-44bf-8f97-e7525a8f513a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b962c19-0a4c-44bf-8f97-e7525a8f513a" (UID: "3b962c19-0a4c-44bf-8f97-e7525a8f513a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:27:53.763856 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.763833    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b962c19-0a4c-44bf-8f97-e7525a8f513a" (UID: "3b962c19-0a4c-44bf-8f97-e7525a8f513a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:27:53.763941 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.763868    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b962c19-0a4c-44bf-8f97-e7525a8f513a-kube-api-access-l758q" (OuterVolumeSpecName: "kube-api-access-l758q") pod "3b962c19-0a4c-44bf-8f97-e7525a8f513a" (UID: "3b962c19-0a4c-44bf-8f97-e7525a8f513a"). InnerVolumeSpecName "kube-api-access-l758q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:27:53.863272 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.863235    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b962c19-0a4c-44bf-8f97-e7525a8f513a-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:53.863272 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.863267    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b962c19-0a4c-44bf-8f97-e7525a8f513a-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:53.863272 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:53.863299    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l758q\" (UniqueName: \"kubernetes.io/projected/3b962c19-0a4c-44bf-8f97-e7525a8f513a-kube-api-access-l758q\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:27:54.625041 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:54.625008    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"
Jul 19 17:27:54.647556 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:54.647529    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"]
Jul 19 17:27:54.653081 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:54.653053    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr"]
Jul 19 17:27:56.615446 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:56.615392    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-8d2863-7f78bcc789-f4pfr" podUID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.73:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 19 17:27:56.632712 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:56.632679    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" path="/var/lib/kubelet/pods/3b962c19-0a4c-44bf-8f97-e7525a8f513a/volumes"
Jul 19 17:27:58.047209 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:58.047169    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:58.058487 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:58.058453    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"]
Jul 19 17:27:58.060556 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:58.060529    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-8d2863-s9nnq"]
Jul 19 17:27:58.518848 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:58.518809    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:27:58.638513 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:27:58.638479    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423728bb-fae5-4d8b-874b-976f6540ae37" path="/var/lib/kubelet/pods/423728bb-fae5-4d8b-874b-976f6540ae37/volumes"
Jul 19 17:28:04.386915 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:04.386878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:04.486928 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:04.486895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:04.785105 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:04.785064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:04.994251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:04.994214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:05.119560 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119478    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"]
Jul 19 17:28:05.119861 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119850    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423728bb-fae5-4d8b-874b-976f6540ae37" containerName="revoke-keys"
Jul 19 17:28:05.119901 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119863    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="423728bb-fae5-4d8b-874b-976f6540ae37" containerName="revoke-keys"
Jul 19 17:28:05.119901 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119880    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" containerName="maas-api"
Jul 19 17:28:05.119901 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119885    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" containerName="maas-api"
Jul 19 17:28:05.119997 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119947    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b962c19-0a4c-44bf-8f97-e7525a8f513a" containerName="maas-api"
Jul 19 17:28:05.119997 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.119956    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="423728bb-fae5-4d8b-874b-976f6540ae37" containerName="revoke-keys"
Jul 19 17:28:05.124209 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.124188    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:05.126970 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.126931    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2254c\""
Jul 19 17:28:05.129135 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.129110    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"]
Jul 19 17:28:05.166895 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.166597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec1af92-b143-48c3-b978-989f49d6e8a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7\" (UID: \"cec1af92-b143-48c3-b978-989f49d6e8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:05.267134 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.267086    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec1af92-b143-48c3-b978-989f49d6e8a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7\" (UID: \"cec1af92-b143-48c3-b978-989f49d6e8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:05.267789 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.267766    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec1af92-b143-48c3-b978-989f49d6e8a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7\" (UID: \"cec1af92-b143-48c3-b978-989f49d6e8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:05.437074 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.437037    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:05.588779 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.588746    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"]
Jul 19 17:28:05.590662 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:28:05.590636    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec1af92_b143_48c3_b978_989f49d6e8a9.slice/crio-c1c4e6cea47ef2e07be89638b849a30b71b29e95dfbd8f601a6991bd2dec3cc4 WatchSource:0}: Error finding container c1c4e6cea47ef2e07be89638b849a30b71b29e95dfbd8f601a6991bd2dec3cc4: Status 404 returned error can't find the container with id c1c4e6cea47ef2e07be89638b849a30b71b29e95dfbd8f601a6991bd2dec3cc4
Jul 19 17:28:05.684348 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.684307    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7" event={"ID":"cec1af92-b143-48c3-b978-989f49d6e8a9","Type":"ContainerStarted","Data":"c1c4e6cea47ef2e07be89638b849a30b71b29e95dfbd8f601a6991bd2dec3cc4"}
Jul 19 17:28:05.786379 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.786308    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:05.887730 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:05.887696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:06.691589 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:06.691555    2584 generic.go:356] "Generic (PLEG): container finished" podID="cec1af92-b143-48c3-b978-989f49d6e8a9" containerID="6c921b7450003c84d1f4fbaf2d9073edfd9098ddb49ce50c65770564b4a38bf0" exitCode=0
Jul 19 17:28:06.692010 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:06.691645    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7" event={"ID":"cec1af92-b143-48c3-b978-989f49d6e8a9","Type":"ContainerDied","Data":"6c921b7450003c84d1f4fbaf2d9073edfd9098ddb49ce50c65770564b4a38bf0"}
Jul 19 17:28:07.838388 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:07.838357    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:07.889796 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:07.889765    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec1af92-b143-48c3-b978-989f49d6e8a9-maas-api-service-ca\") pod \"cec1af92-b143-48c3-b978-989f49d6e8a9\" (UID: \"cec1af92-b143-48c3-b978-989f49d6e8a9\") "
Jul 19 17:28:07.890217 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:07.890180    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec1af92-b143-48c3-b978-989f49d6e8a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cec1af92-b143-48c3-b978-989f49d6e8a9" (UID: "cec1af92-b143-48c3-b978-989f49d6e8a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:28:07.990649 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:07.990567    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cec1af92-b143-48c3-b978-989f49d6e8a9-maas-api-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:28:08.703573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:08.703434    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7" event={"ID":"cec1af92-b143-48c3-b978-989f49d6e8a9","Type":"ContainerDied","Data":"c1c4e6cea47ef2e07be89638b849a30b71b29e95dfbd8f601a6991bd2dec3cc4"}
Jul 19 17:28:08.703573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:08.703487    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1c4e6cea47ef2e07be89638b849a30b71b29e95dfbd8f601a6991bd2dec3cc4"
Jul 19 17:28:08.703573 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:08.703448    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"
Jul 19 17:28:10.788771 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:10.788743    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:28:10.794734 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:10.794712    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:28:11.110170 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:11.110089    2584 scope.go:117] "RemoveContainer" containerID="9aba380302518ee3c1b22d0ce1507728f04f1bdf9119d963a3dcbe39308fce03"
Jul 19 17:28:15.495338 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.495301    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"]
Jul 19 17:28:15.495869 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.495615    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" podUID="942884a3-1bc5-4546-9987-e0c627765e4b" containerName="maas-api" containerID="cri-o://ba992f7197945927702ab17ee13342c0e3920f216121713cc11aeaaadd843c7e" gracePeriod=30
Jul 19 17:28:15.520121 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.520091    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:15.662170 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.662130    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" podUID="942884a3-1bc5-4546-9987-e0c627765e4b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.74:8443/health\": dial tcp 10.133.0.74:8443: connect: connection refused"
Jul 19 17:28:15.739358 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.739331    2584 generic.go:356] "Generic (PLEG): container finished" podID="942884a3-1bc5-4546-9987-e0c627765e4b" containerID="ba992f7197945927702ab17ee13342c0e3920f216121713cc11aeaaadd843c7e" exitCode=0
Jul 19 17:28:15.739471 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.739421    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" event={"ID":"942884a3-1bc5-4546-9987-e0c627765e4b","Type":"ContainerDied","Data":"ba992f7197945927702ab17ee13342c0e3920f216121713cc11aeaaadd843c7e"}
Jul 19 17:28:15.754174 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.754114    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:28:15.862410 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.862373    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942884a3-1bc5-4546-9987-e0c627765e4b-openshift-service-ca\") pod \"942884a3-1bc5-4546-9987-e0c627765e4b\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") "
Jul 19 17:28:15.862629 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.862447    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942884a3-1bc5-4546-9987-e0c627765e4b-maas-api-tls\") pod \"942884a3-1bc5-4546-9987-e0c627765e4b\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") "
Jul 19 17:28:15.862629 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.862494    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znck8\" (UniqueName: \"kubernetes.io/projected/942884a3-1bc5-4546-9987-e0c627765e4b-kube-api-access-znck8\") pod \"942884a3-1bc5-4546-9987-e0c627765e4b\" (UID: \"942884a3-1bc5-4546-9987-e0c627765e4b\") "
Jul 19 17:28:15.862796 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.862766    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942884a3-1bc5-4546-9987-e0c627765e4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "942884a3-1bc5-4546-9987-e0c627765e4b" (UID: "942884a3-1bc5-4546-9987-e0c627765e4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 17:28:15.864851 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.864825    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942884a3-1bc5-4546-9987-e0c627765e4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "942884a3-1bc5-4546-9987-e0c627765e4b" (UID: "942884a3-1bc5-4546-9987-e0c627765e4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 17:28:15.864851 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.864833    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942884a3-1bc5-4546-9987-e0c627765e4b-kube-api-access-znck8" (OuterVolumeSpecName: "kube-api-access-znck8") pod "942884a3-1bc5-4546-9987-e0c627765e4b" (UID: "942884a3-1bc5-4546-9987-e0c627765e4b"). InnerVolumeSpecName "kube-api-access-znck8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 17:28:15.963526 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.963487    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942884a3-1bc5-4546-9987-e0c627765e4b-openshift-service-ca\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:28:15.963526 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.963518    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942884a3-1bc5-4546-9987-e0c627765e4b-maas-api-tls\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:28:15.963526 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:15.963527    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znck8\" (UniqueName: \"kubernetes.io/projected/942884a3-1bc5-4546-9987-e0c627765e4b-kube-api-access-znck8\") on node \"ip-10-0-142-78.ec2.internal\" DevicePath \"\""
Jul 19 17:28:16.745581 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:16.745552    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"
Jul 19 17:28:16.746022 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:16.745586    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn" event={"ID":"942884a3-1bc5-4546-9987-e0c627765e4b","Type":"ContainerDied","Data":"2a9e2a7a864f5546b1a5febbc0a546a2e56d452ac3e04bb74b94b51013511598"}
Jul 19 17:28:16.746022 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:16.745641    2584 scope.go:117] "RemoveContainer" containerID="ba992f7197945927702ab17ee13342c0e3920f216121713cc11aeaaadd843c7e"
Jul 19 17:28:16.761152 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:16.761123    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"]
Jul 19 17:28:16.764518 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:16.764496    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0fa4bd-57584fd8bb-plhqn"]
Jul 19 17:28:18.632786 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:18.632750    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942884a3-1bc5-4546-9987-e0c627765e4b" path="/var/lib/kubelet/pods/942884a3-1bc5-4546-9987-e0c627765e4b/volumes"
Jul 19 17:28:20.192859 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:20.192813    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"]
Jul 19 17:28:20.193357 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:20.193333    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:20.194554 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:20.194535    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0fa4bd-qmtd7"]
Jul 19 17:28:20.629466 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:20.629372    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:20.633231 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:20.633206    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec1af92-b143-48c3-b978-989f49d6e8a9" path="/var/lib/kubelet/pods/cec1af92-b143-48c3-b978-989f49d6e8a9/volumes"
Jul 19 17:28:26.402480 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:26.402443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:26.516589 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:26.516554    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:26.719474 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:26.719373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:26.970898 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:26.970811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:33.216650 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.216622    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-txgvt_542eaa8b-5b41-431b-98a3-f47182937dba/manager/0.log"
Jul 19 17:28:33.350630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.350566    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2hmq5_3ebd0261-2822-4192-9050-03e2f99975c8/manager/0.log"
Jul 19 17:28:33.487181 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.487100    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645d86df7-srxjv_a464873e-dca1-470f-8258-c35484fea26c/manager/0.log"
Jul 19 17:28:33.606909 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.606882    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7mcg4_65c44697-b060-45ad-8346-c502b291e9ac/server/0.log"
Jul 19 17:28:33.748882 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.748801    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nvpc8_49314008-516a-4d6b-a1b6-997f61bdfa44/manager/0.log"
Jul 19 17:28:33.866681 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.866653    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f7db45576-9l62l_2d5007b6-c0f3-40c1-bfb3-7cd5af81168b/copy-manifests/0.log"
Jul 19 17:28:33.880027 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:33.879999    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f7db45576-9l62l_2d5007b6-c0f3-40c1-bfb3-7cd5af81168b/manager/0.log"
Jul 19 17:28:34.542297 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:34.542257    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84f575779b-9tbbr_ea3780ba-bbbe-4a1e-9c81-cfe7c8c37251/maas-api/0.log"
Jul 19 17:28:34.657100 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:34.657072    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741340-htmms_153ba808-4d37-4b91-9196-cab379fc5a54/cleanup/0.log"
Jul 19 17:28:34.773579 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:34.773548    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741355-gztfk_c24f482e-8a01-4b25-8293-f7d12a74a1e8/cleanup/0.log"
Jul 19 17:28:34.897304 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:34.897203    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-7vjxm_5f5c12d7-f2c5-45fe-85c8-eca5979713c9/postgres/0.log"
Jul 19 17:28:36.437730 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:36.437696    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gtq29_5f34d589-30e8-4149-a2b9-3b63964aa9dd/manager/0.log"
Jul 19 17:28:36.560040 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:36.560009    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fb4zt_c49a1761-a9a9-46d9-8d25-41b7318d2b49/manager/0.log"
Jul 19 17:28:36.804681 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:36.804599    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7rtxz_a2841520-adfd-456a-8e78-48251f084716/registry-server/0.log"
Jul 19 17:28:36.945266 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:36.945219    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-799fl_0a897d7c-a6eb-4642-8cc9-a25380274d92/manager/0.log"
Jul 19 17:28:37.206403 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:37.206371    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l8bdr_8840e359-eab4-4c53-9290-d88dab31c522/manager/0.log"
Jul 19 17:28:37.949897 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:37.949867    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdf6466d9-t8sbq_dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2/kube-auth-proxy/0.log"
Jul 19 17:28:38.204192 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:38.204107    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xr6lv_bf6e3017-6988-461f-ab28-4e568492b12a/payload-pre-processing/0.log"
Jul 19 17:28:38.329518 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:38.329483    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-424xr_ecf689aa-0c99-4ad0-8b43-68f002aebebe/payload-processing/0.log"
Jul 19 17:28:39.052041 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:39.052007    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qhq8h_04b84d58-4032-4de6-888a-79e68ccce32f/main/0.log"
Jul 19 17:28:39.414500 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:39.414474    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-qk2fp_86c7d85d-d47d-46ab-94cc-95a96e858e43/main/0.log"
Jul 19 17:28:45.712116 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:45.712081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:45.815346 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:45.815306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log" limit=1024
Jul 19 17:28:48.421444 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:48.421410    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pmwch_c988c3e2-f0e0-4c7e-a826-5ff128caa1ab/global-pull-secret-syncer/0.log"
Jul 19 17:28:48.547579 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:48.547549    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-24cj4_47cb1017-bf16-49de-acdc-4f766a1845cc/konnectivity-agent/0.log"
Jul 19 17:28:48.673378 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:48.673292    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-78.ec2.internal_8bc2a20e77f7595ccd462b1cae835411/haproxy/0.log"
Jul 19 17:28:53.030370 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:53.030341    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-gtq29_5f34d589-30e8-4149-a2b9-3b63964aa9dd/manager/0.log"
Jul 19 17:28:53.052251 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:53.052212    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-fb4zt_c49a1761-a9a9-46d9-8d25-41b7318d2b49/manager/0.log"
Jul 19 17:28:53.144110 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:53.144076    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7rtxz_a2841520-adfd-456a-8e78-48251f084716/registry-server/0.log"
Jul 19 17:28:53.264297 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:53.264255    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-799fl_0a897d7c-a6eb-4642-8cc9-a25380274d92/manager/0.log"
Jul 19 17:28:53.354648 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:53.354550    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-l8bdr_8840e359-eab4-4c53-9290-d88dab31c522/manager/0.log"
Jul 19 17:28:55.293627 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:55.293593    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7ddfd74fc-4nnhk_932cc83c-0424-4040-a701-e456d4186fcc/metrics-server/0.log"
Jul 19 17:28:55.438461 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:55.438426    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9qfx_8a8d1736-1ef0-428e-866b-3f8f796c4471/node-exporter/0.log"
Jul 19 17:28:55.476029 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:55.476004    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9qfx_8a8d1736-1ef0-428e-866b-3f8f796c4471/kube-rbac-proxy/0.log"
Jul 19 17:28:55.494819 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:55.494794    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l9qfx_8a8d1736-1ef0-428e-866b-3f8f796c4471/init-textfile/0.log"
Jul 19 17:28:58.233804 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.233768    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"]
Jul 19 17:28:58.234327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.234170    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec1af92-b143-48c3-b978-989f49d6e8a9" containerName="revoke-keys"
Jul 19 17:28:58.234327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.234182    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec1af92-b143-48c3-b978-989f49d6e8a9" containerName="revoke-keys"
Jul 19 17:28:58.234327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.234199    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942884a3-1bc5-4546-9987-e0c627765e4b" containerName="maas-api"
Jul 19 17:28:58.234327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.234206    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="942884a3-1bc5-4546-9987-e0c627765e4b" containerName="maas-api"
Jul 19 17:28:58.234327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.234266    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="942884a3-1bc5-4546-9987-e0c627765e4b" containerName="maas-api"
Jul 19 17:28:58.234327 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.234304    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec1af92-b143-48c3-b978-989f49d6e8a9" containerName="revoke-keys"
Jul 19 17:28:58.237354 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.237338    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.239690 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.239668    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6jsn\"/\"kube-root-ca.crt\""
Jul 19 17:28:58.239690 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.239680    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6jsn\"/\"openshift-service-ca.crt\""
Jul 19 17:28:58.240731 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.240712    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m6jsn\"/\"default-dockercfg-xgbgc\""
Jul 19 17:28:58.246329 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.246304    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"]
Jul 19 17:28:58.335124 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.335082    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-proc\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.335124 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.335125    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4zn\" (UniqueName: \"kubernetes.io/projected/32612442-d5db-411b-9f82-fdccbbda96ce-kube-api-access-jl4zn\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.335378 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.335163    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-sys\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.335378 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.335237    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-podres\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.335378 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.335274    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-lib-modules\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436386 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436352    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-sys\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436412    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-podres\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436449    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-lib-modules\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436473    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-proc\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4zn\" (UniqueName: \"kubernetes.io/projected/32612442-d5db-411b-9f82-fdccbbda96ce-kube-api-access-jl4zn\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436500    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-sys\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436577 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-proc\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436826 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436625    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-podres\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.436826 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.436634    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32612442-d5db-411b-9f82-fdccbbda96ce-lib-modules\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.444464 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.444439    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4zn\" (UniqueName: \"kubernetes.io/projected/32612442-d5db-411b-9f82-fdccbbda96ce-kube-api-access-jl4zn\") pod \"perf-node-gather-daemonset-kgckz\" (UID: \"32612442-d5db-411b-9f82-fdccbbda96ce\") " pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.548360 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.548252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.692987 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.692960    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"]
Jul 19 17:28:58.695512 ip-10-0-142-78 kubenswrapper[2584]: W0719 17:28:58.695487    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32612442_d5db_411b_9f82_fdccbbda96ce.slice/crio-c3206c4c422d64e9f3f017ee4f6d3b5b9b8d994a816ab4c248ef2d1e7b0acc82 WatchSource:0}: Error finding container c3206c4c422d64e9f3f017ee4f6d3b5b9b8d994a816ab4c248ef2d1e7b0acc82: Status 404 returned error can't find the container with id c3206c4c422d64e9f3f017ee4f6d3b5b9b8d994a816ab4c248ef2d1e7b0acc82
Jul 19 17:28:58.960590 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.960551    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz" event={"ID":"32612442-d5db-411b-9f82-fdccbbda96ce","Type":"ContainerStarted","Data":"6d50c61bbf01dd6c4818ebf844da9ba46e43bfd18e08c276601ef31d7309075c"}
Jul 19 17:28:58.960783 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.960611    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz" event={"ID":"32612442-d5db-411b-9f82-fdccbbda96ce","Type":"ContainerStarted","Data":"c3206c4c422d64e9f3f017ee4f6d3b5b9b8d994a816ab4c248ef2d1e7b0acc82"}
Jul 19 17:28:58.960783 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.960638    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:28:58.978587 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:28:58.978526    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz" podStartSLOduration=0.978509958 podStartE2EDuration="978.509958ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 17:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 17:28:58.97710117 +0000 UTC m=+3048.882352595" watchObservedRunningTime="2026-07-19 17:28:58.978509958 +0000 UTC m=+3048.883761381"
Jul 19 17:29:00.233505 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:00.233476    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcgfq_89cdf330-d376-49a0-9009-1de1be38456a/dns/0.log"
Jul 19 17:29:00.252207 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:00.252177    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcgfq_89cdf330-d376-49a0-9009-1de1be38456a/kube-rbac-proxy/0.log"
Jul 19 17:29:00.321642 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:00.321612    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2g5qt_df49f583-c7e0-4367-af64-995e7bccf5ec/dns-node-resolver/0.log"
Jul 19 17:29:00.870361 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:00.870325    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67ffb6bd68-nwxns_c0703ff3-025a-49ee-91a4-2a8b64334a87/registry/0.log"
Jul 19 17:29:00.914630 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:00.914601    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qbv9q_900bb980-a550-45b4-a0ee-06d3e4c3f2d4/node-ca/0.log"
Jul 19 17:29:01.984538 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:01.984470    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdf6466d9-t8sbq_dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2/kube-auth-proxy/0.log"
Jul 19 17:29:02.063216 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:02.063177    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xr6lv_bf6e3017-6988-461f-ab28-4e568492b12a/payload-pre-processing/0.log"
Jul 19 17:29:02.087906 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:02.087874    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-424xr_ecf689aa-0c99-4ad0-8b43-68f002aebebe/payload-processing/0.log"
Jul 19 17:29:02.761189 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:02.761162    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cg26t_0a8fa31d-fd58-4871-8a13-8fcfb27adcbb/serve-healthcheck-canary/0.log"
Jul 19 17:29:03.402405 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:03.402378    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pbbms_7dfa6b2b-b60a-4c4b-8585-083e98bf4204/kube-rbac-proxy/0.log"
Jul 19 17:29:03.421355 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:03.421333    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pbbms_7dfa6b2b-b60a-4c4b-8585-083e98bf4204/exporter/0.log"
Jul 19 17:29:03.441064 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:03.441031    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pbbms_7dfa6b2b-b60a-4c4b-8585-083e98bf4204/extractor/0.log"
Jul 19 17:29:04.977627 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:04.977601    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m6jsn/perf-node-gather-daemonset-kgckz"
Jul 19 17:29:05.411510 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.411478    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-txgvt_542eaa8b-5b41-431b-98a3-f47182937dba/manager/0.log"
Jul 19 17:29:05.605640 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.605590    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2hmq5_3ebd0261-2822-4192-9050-03e2f99975c8/manager/0.log"
Jul 19 17:29:05.717619 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.717530    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-645d86df7-srxjv_a464873e-dca1-470f-8258-c35484fea26c/manager/0.log"
Jul 19 17:29:05.735775 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.735738    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7mcg4_65c44697-b060-45ad-8346-c502b291e9ac/server/0.log"
Jul 19 17:29:05.804810 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.804779    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-nvpc8_49314008-516a-4d6b-a1b6-997f61bdfa44/manager/0.log"
Jul 19 17:29:05.880386 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.880340    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f7db45576-9l62l_2d5007b6-c0f3-40c1-bfb3-7cd5af81168b/manager/0.log"
Jul 19 17:29:05.902792 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:05.902764    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f7db45576-9l62l_2d5007b6-c0f3-40c1-bfb3-7cd5af81168b/copy-manifests/0.log"
Jul 19 17:29:07.290469 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:07.290441    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fmjbt_c71c323d-89bf-4708-bf38-95f2cd40efa9/openshift-lws-operator/0.log"
Jul 19 17:29:08.311488 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:08.311405    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdf6466d9-t8sbq_dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2/kube-auth-proxy/0.log"
Jul 19 17:29:08.382519 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:08.382485    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xr6lv_bf6e3017-6988-461f-ab28-4e568492b12a/payload-pre-processing/0.log"
Jul 19 17:29:08.402352 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:08.402324    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-424xr_ecf689aa-0c99-4ad0-8b43-68f002aebebe/payload-processing/0.log"
Jul 19 17:29:09.632759 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:09.632667    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cdf6466d9-t8sbq_dcfea2f8-69f5-4901-8eab-ac3b9cf5c4e2/kube-auth-proxy/0.log"
Jul 19 17:29:09.701231 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:09.701201    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xr6lv_bf6e3017-6988-461f-ab28-4e568492b12a/payload-pre-processing/0.log"
Jul 19 17:29:09.719925 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:09.719899    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-424xr_ecf689aa-0c99-4ad0-8b43-68f002aebebe/payload-processing/0.log"
Jul 19 17:29:13.753077 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.753042    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/kube-multus-additional-cni-plugins/0.log"
Jul 19 17:29:13.771954 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.771923    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/egress-router-binary-copy/0.log"
Jul 19 17:29:13.790043 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.790019    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/cni-plugins/0.log"
Jul 19 17:29:13.808185 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.808160    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/bond-cni-plugin/0.log"
Jul 19 17:29:13.827321 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.827267    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/routeoverride-cni/0.log"
Jul 19 17:29:13.847600 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.847564    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/whereabouts-cni-bincopy/0.log"
Jul 19 17:29:13.866904 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.866873    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xrzwf_79070897-558c-4406-a143-39cf00b7a155/whereabouts-cni/0.log"
Jul 19 17:29:13.909039 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:13.909016    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mbbtj_c013d9b4-d2e8-4ff6-9d38-bf57aa458a6d/kube-multus/0.log"
Jul 19 17:29:14.071660 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.071574    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xqq8r_3c09c151-0fa7-4b87-98b9-7b5cf9aedeff/network-metrics-daemon/0.log"
Jul 19 17:29:14.087926 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.087897    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xqq8r_3c09c151-0fa7-4b87-98b9-7b5cf9aedeff/kube-rbac-proxy/0.log"
Jul 19 17:29:14.816970 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.816940    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-controller/0.log"
Jul 19 17:29:14.834247 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.834217    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/1.log"
Jul 19 17:29:14.834865 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.834848    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovn-acl-logging/0.log"
Jul 19 17:29:14.853215 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.853194    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/kube-rbac-proxy-node/0.log"
Jul 19 17:29:14.872063 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.872030    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/kube-rbac-proxy-ovn-metrics/0.log"
Jul 19 17:29:14.887806 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.887766    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/northd/0.log"
Jul 19 17:29:14.906796 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.906771    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/nbdb/0.log"
Jul 19 17:29:14.926117 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.926071    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/sbdb/0.log"
Jul 19 17:29:14.982558 ip-10-0-142-78 kubenswrapper[2584]: I0719 17:29:14.982533    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b4lp2_1b270190-1d31-44c3-9189-ee59c30ad802/ovnkube-controller/0.log"