Jul 22 01:44:47.321564 ip-10-0-130-189 systemd[1]: Starting Kubernetes Kubelet...
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.776374    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781971    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781982    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781986    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781989    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781992    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781996    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.781999    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:44:47.787491 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782002    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782005    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782007    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782011    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782013    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782016    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782019    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782021    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782024    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782027    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782029    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782033    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782036    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782039    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782042    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782048    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782051    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782055    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782057    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782060    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:44:47.803791 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782063    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782065    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782068    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782070    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782073    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782077    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782081    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782084    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782087    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782090    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782093    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782096    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782098    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782101    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782104    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782106    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782108    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782111    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782113    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:44:47.818182 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782116    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782119    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782121    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782123    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782126    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782128    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782130    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782133    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782136    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782139    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782142    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782144    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782147    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782150    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782153    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782155    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782158    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782160    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782163    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782166    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:44:47.839333 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782168    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782171    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782175    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782178    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782181    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782184    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782186    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782189    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782192    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782194    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782197    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782200    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782204    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782206    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782210    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782212    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782215    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782217    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782220    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:44:47.842003 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782223    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782225    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782228    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782230    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782250    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782253    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782256    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782259    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782263    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782267    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782270    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782273    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782275    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782278    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782973    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782979    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782982    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782985    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782987    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782990    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:44:47.844977 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782993    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782995    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.782998    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783000    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783003    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783005    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783008    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783010    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783014    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783017    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783021    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783026    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783029    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783032    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783035    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783037    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783042    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783044    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783047    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:44:47.849173 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783050    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783052    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783057    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783060    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783063    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783067    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783070    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783073    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783075    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783078    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783081    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783085    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783087    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783090    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783092    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783095    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783097    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783100    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783103    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:44:47.854580 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783105    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783108    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783111    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783115    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783117    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783121    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783125    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783127    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783131    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783134    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783137    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783140    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783142    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783145    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783147    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783150    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783152    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783156    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783159    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:44:47.858934 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783161    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:44:47.855822 ip-10-0-130-189 systemd[1]: Started Kubernetes Kubelet.
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783164    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783166    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783169    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783171    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783174    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783176    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783179    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783181    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783184    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783186    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783189    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783192    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783196    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783199    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783201    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783204    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783208    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783210    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783213    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:44:47.868871 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783216    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783218    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783221    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783224    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783226    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783229    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783231    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783250    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783253    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783256    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783258    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783261    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783264    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783266    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783269    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783271    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783648    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783653    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783657    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783660    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:44:47.873330 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783663    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783666    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783669    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783672    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783674    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783679    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783681    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783684    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783687    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783690    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783693    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783695    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783698    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783701    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783703    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783706    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783708    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783711    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783713    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:44:47.881477 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783716    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783718    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783721    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783723    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783726    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783728    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783731    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783734    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783737    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783739    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783742    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783744    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783747    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783749    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783752    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783755    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783757    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783760    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783763    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783766    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:44:47.887453 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783769    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783771    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783774    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783777    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783779    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783782    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783784    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783787    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783790    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783793    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783795    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783797    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783800    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783802    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783805    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783809    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783813    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783816    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783818    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783821    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:44:48.042701 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783824    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783826    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783829    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783831    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783833    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783836    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783839    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783841    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783844    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783846    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783850    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783853    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783856    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783860    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783862    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783865    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783868    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783871    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783874    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783876    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:44:48.064311 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783878    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783881    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783884    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783886    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783890    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783893    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783897    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783901    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783905    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783908    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783911    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783913    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783916    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783919    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783922    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.783924    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.783986    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.783993    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.783998    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784003    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 22 01:44:48.067337 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784007    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784010    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784016    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784020    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784024    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784027    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784030    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784033    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784036    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784041    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784044    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784047    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784050    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784053    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784057    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784060    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784063    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784066    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784069    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784073    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784076    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784079    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784082    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784085    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 22 01:44:48.068834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784088    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784091    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784094    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784097    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784101    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784104    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784107    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784110    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784113    2561 flags.go:64] FLAG: --enable-server="true"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784117    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784124    2561 flags.go:64] FLAG: --event-burst="100"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784127    2561 flags.go:64] FLAG: --event-qps="50"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784130    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784133    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784136    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784140    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784143    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784146    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784150    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784153    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784156    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784159    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784161    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784164    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784167    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 22 01:44:48.070228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784170    2561 flags.go:64] FLAG: --feature-gates=""
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784173    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784176    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784188    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784192    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784195    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784198    2561 flags.go:64] FLAG: --help="false"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784200    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784203    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784208    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784212    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784215    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784218    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784221    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784224    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784227    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784230    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784247    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784250    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784253    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784256    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784259    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784262    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784265    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 22 01:44:48.071215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784268    2561 flags.go:64] FLAG: --lock-file=""
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784270    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784274    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784277    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784283    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784285    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784288    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784291    2561 flags.go:64] FLAG: --logging-format="text"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784294    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784297    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784300    2561 flags.go:64] FLAG: --manifest-url=""
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784303    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784307    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784309    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784313    2561 flags.go:64] FLAG: --max-pods="110"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784316    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784319    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784323    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784326    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784329    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784332    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784335    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784342    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784345    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784348    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 22 01:44:48.072041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784352    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784355    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784362    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784364    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784368    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784371    2561 flags.go:64] FLAG: --port="10250"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784373    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784376    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d6fd45b780d1ebcc"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784379    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784382    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784390    2561 flags.go:64] FLAG: --register-node="true"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784394    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784397    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784401    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784404    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784406    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784410    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784413    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784416    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784419    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784422    2561 flags.go:64] FLAG: --runonce="false"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784424    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784427    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784430    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784434    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 22 01:44:48.072869 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784437    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784440    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784443    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784446    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784449    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784452    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784455    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784458    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784462    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784465    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784468    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784473    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784476    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784479    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784483    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784486    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784489    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784491    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784495    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784498    2561 flags.go:64] FLAG: --v="2"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784507    2561 flags.go:64] FLAG: --version="false"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784511    2561 flags.go:64] FLAG: --vmodule=""
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784514    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784517    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784629    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:44:48.073691 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784633    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784636    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784639    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784642    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784645    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784648    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784651    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784654    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784657    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784660    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784662    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784665    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784667    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784670    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784672    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784676    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784679    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784681    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784684    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:44:48.074475 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784686    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784689    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784691    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784694    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784697    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784699    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784701    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784704    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784709    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784712    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784715    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784718    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784721    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784724    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784726    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784729    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784732    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784735    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784738    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:44:48.075000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784742    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784745    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784747    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784750    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784753    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784756    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784759    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784761    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784764    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784768    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784770    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784773    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784775    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784778    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784781    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784784    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784786    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784789    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784791    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784794    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:44:48.075589 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784797    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784800    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784803    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784805    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784808    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784810    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784813    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784815    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784818    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784820    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784823    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784825    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784831    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784834    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784837    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784840    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784843    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784846    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784848    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:44:48.076458 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784851    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784853    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784857    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784860    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784862    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784865    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784868    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784870    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784873    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784875    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784878    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784880    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784883    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784887    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784890    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784894    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784896    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784899    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784901    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784904    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:44:48.077000 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.784907    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.784912    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.793305    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.793319    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.793339    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.793345    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793400    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793406    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793409    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793413    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793416    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793419    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793421    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793424    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793427    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:44:48.077608 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793430    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793432    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793435    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793438    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793440    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793444    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793448    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793452    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793457    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793460    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793463    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793466    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793469    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793472    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793475    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793477    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793480    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793482    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793485    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:44:48.078086 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793488    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793491    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793494    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793496    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793498    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793501    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793504    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793506    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793509    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793511    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793514    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793518    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793521    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793524    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793527    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793530    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793533    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793536    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793539    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:44:48.078947 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793542    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793545    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793548    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793551    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793554    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793557    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793560    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793563    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793566    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793569    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793572    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793574    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793577    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793580    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793583    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793586    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793588    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793591    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793593    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793596    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:44:48.079781 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793599    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793601    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793604    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793606    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793609    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793612    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793614    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793617    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793619    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793622    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793624    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793627    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793630    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793633    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793636    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793641    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793643    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793646    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793648    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:44:48.080536 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793652    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793654    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793657    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793659    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793662    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793667    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793670    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793672    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793675    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793678    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793680    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793683    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.793688    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793797    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 01:44:48.081118 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793802    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793805    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793808    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793811    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793814    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793817    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793819    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793822    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793825    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793827    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793830    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793834    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793838    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793841    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793844    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793846    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793849    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793851    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793854    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 01:44:48.081725 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793862    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793865    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793867    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793870    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793874    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793877    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793879    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793882    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793884    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793887    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793891    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793893    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793896    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793899    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793901    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793904    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793906    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793909    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793912    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 01:44:48.082694 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793914    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793918    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793921    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793925    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793928    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793930    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793933    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793936    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793939    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793941    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793944    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793947    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793949    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793952    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793955    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793958    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793960    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793967    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793969    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793972    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 01:44:48.083574 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793974    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793976    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793979    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793983    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793986    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793989    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793992    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793995    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.793998    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794001    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794004    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794006    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794009    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794012    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794014    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794017    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794019    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794022    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794025    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794027    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 01:44:48.084479 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794030    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794032    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794035    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794037    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794040    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794042    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794045    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794048    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794050    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794055    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794057    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794060    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794062    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794065    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794067    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794071    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794073    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794076    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794079    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 01:44:48.085722 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:47.794081    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.794086    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.794190    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.796577    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.798288    2561 server.go:1019] "Starting client certificate rotation"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.798389    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.799280    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.801023    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.802331    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.818175    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.820753    2561 log.go:25] "Validated CRI v1 image API"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.821508    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 22 01:44:48.086355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.823987    2561 fs.go:135] Filesystem UUIDs: map[0246929b-bd2d-408f-9595-febc86030f2d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d10cc96a-2a6f-44af-bc6e-0401df3da890:/dev/nvme0n1p4]
Jul 22 01:44:48.086986 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.824006    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.831033    2561 manager.go:216] Machine: {Timestamp:2026-07-22 01:44:47.828671147 +0000 UTC m=+0.377022095 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105937 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cd7c8d913a3163979161a7c17277e SystemUUID:ec2cd7c8-d913-a316-3979-161a7c17277e BootID:4bc996d1-fb7e-4046-b35a-f582261a16c8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b3:05:0d:d7:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b3:05:0d:d7:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:f8:e7:b8:77:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.831125    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.831199    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.831607    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.834668    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.834696    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-189.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 22 01:44:48.087191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.835574    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.835588    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.835598    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.836488    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.838419    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.842993    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843123    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843152    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843168    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843183    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843198    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843218    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843599    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.843619    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.846181    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.848877    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.848906    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850460    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850476    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850483    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850489    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850498    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850524    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850532    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850537    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850546    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850551    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850564    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.850572    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.851397    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.851406    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.854188    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-189.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.854428    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.854433    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-189.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.854929    2561 server.go:1268] "Started kubelet"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.854953    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.855048    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.855107    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.855449    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b8q9f"
Jul 22 01:44:48.087798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.857259    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.862689    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b8q9f"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.863046    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.865664    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.867018    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-189.ec2.internal.18c4794497a7e82c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-189.ec2.internal,UID:ip-10-0-130-189.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-189.ec2.internal,},FirstTimestamp:2026-07-22 01:44:47.854659628 +0000 UTC m=+0.403010577,LastTimestamp:2026-07-22 01:44:47.854659628 +0000 UTC m=+0.403010577,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-189.ec2.internal,}"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.874500    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.874511    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.874623    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875269    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875288    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875338    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875345    2561 factory.go:55] Registering systemd factory
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875360    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875413    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875423    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875554    2561 factory.go:153] Registering CRI-O factory
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875569    2561 factory.go:223] Registration of the crio container factory successfully
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875627    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875648    2561 factory.go:103] Registering Raw factory
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.875661    2561 manager.go:1196] Started watching for new ooms in manager
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.876047    2561 manager.go:349] Starting recovery of all containers
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.884775    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.887129    2561 manager.go:354] Recovery completed
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.887499    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-189.ec2.internal\" not found" node="ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.891731    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.893612    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.893642    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.893656    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.894188    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.894197    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.894212    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.894298    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.895142    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.895167    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:47.895176    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.088833 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:47.975447    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.041390    2561 policy_none.go:49] "None policy: Start"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.041409    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.041421    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.043768    2561 policy_none.go:47] "Start"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.049060    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.050223    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.050294    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.050323    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.050377    2561 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.054751    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.075901    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.078913    2561 manager.go:337] "Starting Device Plugin manager"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.078943    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.078952    2561 server.go:87] "Starting device plugin registration server"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.079223    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.079248    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.079334    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.079410    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.079420    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.079935    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 22 01:44:48.089819 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.079976    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.151158    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"]
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.151267    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.152331    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.152357    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.152367    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.154712    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.154815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.154841    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.155879    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.155887    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.155902    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.155911    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.155916    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.155921    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.158125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.158145    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.159115    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.159145    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.159154    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.180027    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.180888    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.180913    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.180925    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeHasSufficientPID"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.180950    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.185913    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.185928    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-189.ec2.internal\": node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.187852    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-189.ec2.internal\" not found" node="ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.192408    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-189.ec2.internal\" not found" node="ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.200200    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.298156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.276225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e61819b72a47ad3f811b99f43a91c5fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"e61819b72a47ad3f811b99f43a91c5fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.299533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.276268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e61819b72a47ad3f811b99f43a91c5fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"e61819b72a47ad3f811b99f43a91c5fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.299533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.276289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f6a1ecb72d07fd09a4dca395487a6f2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-189.ec2.internal\" (UID: \"1f6a1ecb72d07fd09a4dca395487a6f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.301650 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.301017    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.376394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e61819b72a47ad3f811b99f43a91c5fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"e61819b72a47ad3f811b99f43a91c5fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.376442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e61819b72a47ad3f811b99f43a91c5fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"e61819b72a47ad3f811b99f43a91c5fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.376466    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f6a1ecb72d07fd09a4dca395487a6f2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-189.ec2.internal\" (UID: \"1f6a1ecb72d07fd09a4dca395487a6f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.376487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e61819b72a47ad3f811b99f43a91c5fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"e61819b72a47ad3f811b99f43a91c5fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.376519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1f6a1ecb72d07fd09a4dca395487a6f2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-189.ec2.internal\" (UID: \"1f6a1ecb72d07fd09a4dca395487a6f2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.376564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e61819b72a47ad3f811b99f43a91c5fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal\" (UID: \"e61819b72a47ad3f811b99f43a91c5fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.401502    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.491945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:48.497351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:48.518284 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.502469    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:49.734578 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:48.603204    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:49.749407    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:49.842973    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:49.843132    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:49.843175    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:49.850232    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:49.918003    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 01:39:48 +0000 UTC" deadline="2028-04-10 07:55:03.6654003 +0000 UTC"
Jul 22 01:44:49.924019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:49.918026    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15078h10m13.747376974s"
Jul 22 01:44:50.089315 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:49.950931    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:50.089315 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:50.051410    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:50.089315 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.069634    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:44:50.285514 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:50.151876    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:50.285514 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:50.252357    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-189.ec2.internal\" not found"
Jul 22 01:44:50.285514 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:50.279464    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6a1ecb72d07fd09a4dca395487a6f2.slice/crio-47a9dce38141c2052e6b134c46cf671287b48c2b5b13b1d1015f1ef87a735fa4 WatchSource:0}: Error finding container 47a9dce38141c2052e6b134c46cf671287b48c2b5b13b1d1015f1ef87a735fa4: Status 404 returned error can't find the container with id 47a9dce38141c2052e6b134c46cf671287b48c2b5b13b1d1015f1ef87a735fa4
Jul 22 01:44:50.285514 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:50.280192    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61819b72a47ad3f811b99f43a91c5fa.slice/crio-a3851c51bee468c7c3c3e1af03bae0a3368228eff6b31a985b6470e890281256 WatchSource:0}: Error finding container a3851c51bee468c7c3c3e1af03bae0a3368228eff6b31a985b6470e890281256: Status 404 returned error can't find the container with id a3851c51bee468c7c3c3e1af03bae0a3368228eff6b31a985b6470e890281256
Jul 22 01:44:50.285514 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.285457    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 01:44:50.287554 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.287533    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:44:50.320902 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.320871    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:50.333285 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.333265    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 01:44:50.334788 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.334776    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal"
Jul 22 01:44:50.342930 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.342886    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 01:44:50.837148 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.837049    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:44:50.891506 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.891479    2561 apiserver.go:52] "Watching apiserver"
Jul 22 01:44:50.904329 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.904304    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 22 01:44:50.905119 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.905083    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-rh2rd","openshift-network-operator/iptables-alerter-t9rjn","kube-system/global-pull-secret-syncer-sqzb4","kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n","openshift-cluster-node-tuning-operator/tuned-vxpkx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal","openshift-multus/multus-additional-cni-plugins-v8zbk","openshift-network-diagnostics/network-check-target-wpwxr","openshift-ovn-kubernetes/ovnkube-node-lf44z","kube-system/konnectivity-agent-m4hg5","openshift-dns/node-resolver-t8j9b","openshift-image-registry/node-ca-s77x5","openshift-multus/multus-rnfk6"]
Jul 22 01:44:50.908079 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.908060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:50.910508 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.910486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.910716 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.910698    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 22 01:44:50.910932 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.910914    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wbsj4\""
Jul 22 01:44:50.911579 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.911172    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.911579 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.911177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:50.913232 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.913211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:50.913329 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:50.913272    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:44:50.913607 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.913590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.914243 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.913808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 22 01:44:50.914243 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.913989    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6xmts\""
Jul 22 01:44:50.914243 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.914218    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.915285 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.915266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:50.915372 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:50.915333    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:44:50.920304 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.919907    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.920304 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.920265    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 22 01:44:50.921936 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.921871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.922433 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.922413    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.922554 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.922456    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9th8c\""
Jul 22 01:44:50.922631 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.922569    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.922800 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.922718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 22 01:44:50.924985 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.923897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.924985 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.924243    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:50.925695 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.925407    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v4brs\""
Jul 22 01:44:50.925695 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.925464    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.926661 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.926625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:50.927744 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.927702    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.928040 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.928022    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.928121 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.928082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 22 01:44:50.928315 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.928297    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pwlwk\""
Jul 22 01:44:50.928523 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.928504    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 22 01:44:50.929259 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysconfig\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.929394 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-dbus\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:50.929506 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929320    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-registration-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-sys-fs\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929527    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mclr9\""
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:50.929643    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wx2b\" (UniqueName: \"kubernetes.io/projected/c6dff408-2a12-4984-b95b-8937ad3254da-kube-api-access-6wx2b\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-kubernetes\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysctl-conf\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.929769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-run\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-host\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:50.930279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/773f8125-c875-4892-9e3a-201595bac469-host-slash\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-sys\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-lib-modules\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmksj\" (UniqueName: \"kubernetes.io/projected/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-kube-api-access-qmksj\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/773f8125-c875-4892-9e3a-201595bac469-iptables-alerter-script\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-host\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-systemd\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7a3c44b-41d2-490e-a053-e6319355233b-tmp\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-serviceca\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhj7\" (UniqueName: \"kubernetes.io/projected/6e53db84-b279-44fb-a467-07f335547a01-kube-api-access-4vhj7\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-etc-selinux\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysctl-d\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-kubelet-config\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:50.930972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5fs\" (UniqueName: \"kubernetes.io/projected/f7a3c44b-41d2-490e-a053-e6319355233b-kube-api-access-2n5fs\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvd6\" (UniqueName: \"kubernetes.io/projected/773f8125-c875-4892-9e3a-201595bac469-kube-api-access-8wvd6\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-modprobe-d\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-var-lib-kubelet\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7a3c44b-41d2-490e-a053-e6319355233b-etc-tuned\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-socket-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.931600 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.930800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-device-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:50.932610 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.931945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:50.934264 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.934245    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:50.934687 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.934669    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.934779 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.934742    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.934847 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.934821    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 22 01:44:50.935748 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.935708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcfkq\""
Jul 22 01:44:50.936547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.935941    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 22 01:44:50.936547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.936026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 22 01:44:50.936547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.936126    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 01:44:50.936547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.936162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:50.936547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.936344    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 22 01:44:50.936547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.936455    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 22 01:44:50.938184 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.938163    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-799wc\""
Jul 22 01:44:50.938825 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.938808    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 22 01:44:50.939349 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.939330    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 22 01:44:50.940347 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.940075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qqgh5\""
Jul 22 01:44:50.940791 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.940503    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 22 01:44:50.952933 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.952913    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fvtkj"
Jul 22 01:44:50.960270 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:50.960251    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fvtkj"
Jul 22 01:44:51.022316 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.022286    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 22 01:44:51.031195 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031166    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysctl-d\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.031317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-kubelet-config\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:51.031317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-socket-dir-parent\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.031317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2pk\" (UniqueName: \"kubernetes.io/projected/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-kube-api-access-7h2pk\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.031317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-cni-bin\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.031317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5fs\" (UniqueName: \"kubernetes.io/projected/f7a3c44b-41d2-490e-a053-e6319355233b-kube-api-access-2n5fs\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031339    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-cni-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysctl-d\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-kubelet-config\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-modprobe-d\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031459    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.031496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-socket-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-modprobe-d\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-cni-binary-copy\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-cni-multus\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031604    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-socket-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-cni-netd\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-env-overrides\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:51.031790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-registration-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wx2b\" (UniqueName: \"kubernetes.io/projected/c6dff408-2a12-4984-b95b-8937ad3254da-kube-api-access-6wx2b\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-k8s-cni-cncf-io\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.031831    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-etc-kubernetes\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-os-release\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.031903    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:51.531871107 +0000 UTC m=+3.034729328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.031921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-registration-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-slash\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-ovnkube-config\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysctl-conf\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-host\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-run-netns\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.032191 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/773f8125-c875-4892-9e3a-201595bac469-iptables-alerter-script\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a4e8df-2d58-4fa6-a084-484802fabbbf-tmp-dir\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-host\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgtw\" (UniqueName: \"kubernetes.io/projected/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-kube-api-access-zbgtw\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysctl-conf\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-kubelet\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-systemd-units\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-systemd\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-serviceca\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0f19b6e-6eec-4ad8-af2f-69c5844b7abd-agent-certs\") pod \"konnectivity-agent-m4hg5\" (UID: \"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd\") " pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0f19b6e-6eec-4ad8-af2f-69c5844b7abd-konnectivity-ca\") pod \"konnectivity-agent-m4hg5\" (UID: \"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd\") " pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-cni-bin\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.032843 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032613    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-hostroot\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-daemon-config\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vbn\" (UniqueName: \"kubernetes.io/projected/4eeb6b09-899e-4336-8693-99407d362af3-kube-api-access-95vbn\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-os-release\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-netns\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-etc-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-ovnkube-script-lib\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2a4e8df-2d58-4fa6-a084-484802fabbbf-hosts-file\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlwr\" (UniqueName: \"kubernetes.io/projected/b2a4e8df-2d58-4fa6-a084-484802fabbbf-kube-api-access-2jlwr\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvd6\" (UniqueName: \"kubernetes.io/projected/773f8125-c875-4892-9e3a-201595bac469-kube-api-access-8wvd6\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-systemd\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032913    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-var-lib-kubelet\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7a3c44b-41d2-490e-a053-e6319355233b-etc-tuned\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.033580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-device-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.032987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-system-cni-dir\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-node-log\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysconfig\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-dbus\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-sys-fs\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-kubelet\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-conf-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-multus-certs\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-ovn\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033302    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033347    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-kubernetes\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-run\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/773f8125-c875-4892-9e3a-201595bac469-iptables-alerter-script\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/773f8125-c875-4892-9e3a-201595bac469-host-slash\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-var-lib-kubelet\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-system-cni-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-device-dir\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-systemd\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033582    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-sysconfig\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eeb6b09-899e-4336-8693-99407d362af3-ovn-node-metrics-cert\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/773f8125-c875-4892-9e3a-201595bac469-host-slash\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-sys-fs\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-etc-kubernetes\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-sys\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.034992 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-lib-modules\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033740    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-run\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-dbus\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-sys\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033835    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmksj\" (UniqueName: \"kubernetes.io/projected/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-kube-api-access-qmksj\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-lib-modules\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.033985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-var-lib-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-host\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-serviceca\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7a3c44b-41d2-490e-a053-e6319355233b-host\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-cnibin\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.034196    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:51.035628 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7a3c44b-41d2-490e-a053-e6319355233b-tmp\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.036231 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.034258    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:44:51.534241671 +0000 UTC m=+3.037099893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:51.036231 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhj7\" (UniqueName: \"kubernetes.io/projected/6e53db84-b279-44fb-a467-07f335547a01-kube-api-access-4vhj7\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:51.036231 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-etc-selinux\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.036231 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034402    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cnibin\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.036231 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-log-socket\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.036231 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.034524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6dff408-2a12-4984-b95b-8937ad3254da-etc-selinux\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.037228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.037207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7a3c44b-41d2-490e-a053-e6319355233b-etc-tuned\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.037331 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.037252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7a3c44b-41d2-490e-a053-e6319355233b-tmp\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.041161 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.041131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5fs\" (UniqueName: \"kubernetes.io/projected/f7a3c44b-41d2-490e-a053-e6319355233b-kube-api-access-2n5fs\") pod \"tuned-vxpkx\" (UID: \"f7a3c44b-41d2-490e-a053-e6319355233b\") " pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.041261 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.041189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvd6\" (UniqueName: \"kubernetes.io/projected/773f8125-c875-4892-9e3a-201595bac469-kube-api-access-8wvd6\") pod \"iptables-alerter-t9rjn\" (UID: \"773f8125-c875-4892-9e3a-201595bac469\") " pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.041348 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.041328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wx2b\" (UniqueName: \"kubernetes.io/projected/c6dff408-2a12-4984-b95b-8937ad3254da-kube-api-access-6wx2b\") pod \"aws-ebs-csi-driver-node-gh52n\" (UID: \"c6dff408-2a12-4984-b95b-8937ad3254da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.041407 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.041331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmksj\" (UniqueName: \"kubernetes.io/projected/cab3a42b-231d-4f4e-bbec-1ed8e9421ecc-kube-api-access-qmksj\") pod \"node-ca-s77x5\" (UID: \"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc\") " pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.044483 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.044462    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhj7\" (UniqueName: \"kubernetes.io/projected/6e53db84-b279-44fb-a467-07f335547a01-kube-api-access-4vhj7\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:51.057994 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.057974    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 01:44:51.101244 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.101148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" event={"ID":"e61819b72a47ad3f811b99f43a91c5fa","Type":"ContainerStarted","Data":"a3851c51bee468c7c3c3e1af03bae0a3368228eff6b31a985b6470e890281256"}
Jul 22 01:44:51.102351 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.102323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal" event={"ID":"1f6a1ecb72d07fd09a4dca395487a6f2","Type":"ContainerStarted","Data":"47a9dce38141c2052e6b134c46cf671287b48c2b5b13b1d1015f1ef87a735fa4"}
Jul 22 01:44:51.135247 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-k8s-cni-cncf-io\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.135247 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-etc-kubernetes\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-os-release\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-slash\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-k8s-cni-cncf-io\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-ovnkube-config\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-slash\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-etc-kubernetes\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-os-release\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.135445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-run-netns\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.135775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135451    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-run-netns\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.135775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135453    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a4e8df-2d58-4fa6-a084-484802fabbbf-tmp-dir\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.135775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgtw\" (UniqueName: \"kubernetes.io/projected/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-kube-api-access-zbgtw\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.135775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.135775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a4e8df-2d58-4fa6-a084-484802fabbbf-tmp-dir\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.135775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-kubelet\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135779    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-systemd-units\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135799    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-kubelet\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135870    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-systemd-units\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-run-ovn-kubernetes\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0f19b6e-6eec-4ad8-af2f-69c5844b7abd-agent-certs\") pod \"konnectivity-agent-m4hg5\" (UID: \"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd\") " pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.135959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0f19b6e-6eec-4ad8-af2f-69c5844b7abd-konnectivity-ca\") pod \"konnectivity-agent-m4hg5\" (UID: \"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd\") " pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.136051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-ovnkube-config\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-cni-bin\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-hostroot\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136149    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-daemon-config\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cni-binary-copy\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136168    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vbn\" (UniqueName: \"kubernetes.io/projected/4eeb6b09-899e-4336-8693-99407d362af3-kube-api-access-95vbn\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-cni-bin\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-hostroot\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-os-release\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-netns\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-etc-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136364    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-netns\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-ovnkube-script-lib\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2a4e8df-2d58-4fa6-a084-484802fabbbf-hosts-file\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-etc-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0f19b6e-6eec-4ad8-af2f-69c5844b7abd-konnectivity-ca\") pod \"konnectivity-agent-m4hg5\" (UID: \"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd\") " pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlwr\" (UniqueName: \"kubernetes.io/projected/b2a4e8df-2d58-4fa6-a084-484802fabbbf-kube-api-access-2jlwr\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2a4e8df-2d58-4fa6-a084-484802fabbbf-hosts-file\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-os-release\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-systemd\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-system-cni-dir\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-node-log\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-kubelet\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.136836 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-system-cni-dir\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-conf-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-multus-certs\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-conf-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-ovn\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-ovn\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-node-log\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136766    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-systemd\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-system-cni-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-run-multus-certs\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-daemon-config\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.137564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136843    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eeb6b09-899e-4336-8693-99407d362af3-ovn-node-metrics-cert\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-system-cni-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-var-lib-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-kubelet\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-var-lib-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.136996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-ovnkube-script-lib\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-cnibin\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cnibin\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-log-socket\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-socket-dir-parent\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-cnibin\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2pk\" (UniqueName: \"kubernetes.io/projected/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-kube-api-access-7h2pk\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-cni-bin\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-cni-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137165    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-log-socket\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-cni-bin\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-cnibin\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-cni-binary-copy\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-cni-multus\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-cni-dir\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137325    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-cni-netd\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-multus-socket-dir-parent\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-env-overrides\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137358    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-host-cni-netd\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eeb6b09-899e-4336-8693-99407d362af3-run-openvswitch\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.138895 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-host-var-lib-cni-multus\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.139439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-cni-binary-copy\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.139439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.137853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eeb6b09-899e-4336-8693-99407d362af3-env-overrides\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.139439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.139015    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0f19b6e-6eec-4ad8-af2f-69c5844b7abd-agent-certs\") pod \"konnectivity-agent-m4hg5\" (UID: \"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd\") " pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.139439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.139391    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eeb6b09-899e-4336-8693-99407d362af3-ovn-node-metrics-cert\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.143342 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.142956    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:44:51.143342 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.143010    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:44:51.143342 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.143025    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:51.143342 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.143097    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:51.643078448 +0000 UTC m=+3.145936690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:51.143879 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.143857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgtw\" (UniqueName: \"kubernetes.io/projected/64c2937f-17fb-4aa5-881e-c859d6f9ae0a-kube-api-access-zbgtw\") pod \"multus-rnfk6\" (UID: \"64c2937f-17fb-4aa5-881e-c859d6f9ae0a\") " pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.144324 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.144299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vbn\" (UniqueName: \"kubernetes.io/projected/4eeb6b09-899e-4336-8693-99407d362af3-kube-api-access-95vbn\") pod \"ovnkube-node-lf44z\" (UID: \"4eeb6b09-899e-4336-8693-99407d362af3\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.145195 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.145179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2pk\" (UniqueName: \"kubernetes.io/projected/3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7-kube-api-access-7h2pk\") pod \"multus-additional-cni-plugins-v8zbk\" (UID: \"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7\") " pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.145484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.145464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlwr\" (UniqueName: \"kubernetes.io/projected/b2a4e8df-2d58-4fa6-a084-484802fabbbf-kube-api-access-2jlwr\") pod \"node-resolver-t8j9b\" (UID: \"b2a4e8df-2d58-4fa6-a084-484802fabbbf\") " pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.222504 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.222463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s77x5"
Jul 22 01:44:51.232389 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.232368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t9rjn"
Jul 22 01:44:51.248294 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.248267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n"
Jul 22 01:44:51.254118 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.254097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vxpkx"
Jul 22 01:44:51.262819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.262788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnfk6"
Jul 22 01:44:51.274358 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.274339    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v8zbk"
Jul 22 01:44:51.281020 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.280999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:44:51.289630 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.289611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:44:51.296212 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.296191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t8j9b"
Jul 22 01:44:51.540228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.540138    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:51.540228 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.540190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:51.540449 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.540286    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:51.540449 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.540290    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:51.540449 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.540349    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:52.540325815 +0000 UTC m=+4.043184026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:51.540449 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.540367    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:44:52.540357347 +0000 UTC m=+4.043215557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:51.741986 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.741953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:51.742138 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.742094    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:44:51.742138 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.742109    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:44:51.742138 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.742120    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:51.742277 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:51.742174    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:52.742156847 +0000 UTC m=+4.245015074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:51.963561 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.963468    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:39:50 +0000 UTC" deadline="2028-02-22 17:46:52.434890842 +0000 UTC"
Jul 22 01:44:51.963561 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:51.963513    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h2m0.471389857s"
Jul 22 01:44:52.082228 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.082194    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c2937f_17fb_4aa5_881e_c859d6f9ae0a.slice/crio-efced52d96df9ea717ed7ea908ddc2d05a40ef5282682e7b262381360756dfbb WatchSource:0}: Error finding container efced52d96df9ea717ed7ea908ddc2d05a40ef5282682e7b262381360756dfbb: Status 404 returned error can't find the container with id efced52d96df9ea717ed7ea908ddc2d05a40ef5282682e7b262381360756dfbb
Jul 22 01:44:52.084153 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.084127    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a3c44b_41d2_490e_a053_e6319355233b.slice/crio-f7062f3980698e3af1f597f8b4ee3ba0e304d945873d990c750828283d87536c WatchSource:0}: Error finding container f7062f3980698e3af1f597f8b4ee3ba0e304d945873d990c750828283d87536c: Status 404 returned error can't find the container with id f7062f3980698e3af1f597f8b4ee3ba0e304d945873d990c750828283d87536c
Jul 22 01:44:52.085168 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.085061    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f19b6e_6eec_4ad8_af2f_69c5844b7abd.slice/crio-24e0cd8caf892edadf00a3726e0bb70c53a4247a157a58a706c46b28e5adcb19 WatchSource:0}: Error finding container 24e0cd8caf892edadf00a3726e0bb70c53a4247a157a58a706c46b28e5adcb19: Status 404 returned error can't find the container with id 24e0cd8caf892edadf00a3726e0bb70c53a4247a157a58a706c46b28e5adcb19
Jul 22 01:44:52.087961 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.087937    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a4e8df_2d58_4fa6_a084_484802fabbbf.slice/crio-686fed6050c7fe40febdbf848f81ac6e059e474841b3fde4e46394b50b5189b1 WatchSource:0}: Error finding container 686fed6050c7fe40febdbf848f81ac6e059e474841b3fde4e46394b50b5189b1: Status 404 returned error can't find the container with id 686fed6050c7fe40febdbf848f81ac6e059e474841b3fde4e46394b50b5189b1
Jul 22 01:44:52.088742 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.088706    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab3a42b_231d_4f4e_bbec_1ed8e9421ecc.slice/crio-3f0606cd7fbe20ae3be67752691242f1da5b33865425346d110612a94cdb703d WatchSource:0}: Error finding container 3f0606cd7fbe20ae3be67752691242f1da5b33865425346d110612a94cdb703d: Status 404 returned error can't find the container with id 3f0606cd7fbe20ae3be67752691242f1da5b33865425346d110612a94cdb703d
Jul 22 01:44:52.089908 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.089503    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6dff408_2a12_4984_b95b_8937ad3254da.slice/crio-7390884b9b08de5f1eac528fea8523f45d5a76ee1ca1f8777fd525d88d5878f3 WatchSource:0}: Error finding container 7390884b9b08de5f1eac528fea8523f45d5a76ee1ca1f8777fd525d88d5878f3: Status 404 returned error can't find the container with id 7390884b9b08de5f1eac528fea8523f45d5a76ee1ca1f8777fd525d88d5878f3
Jul 22 01:44:52.090382 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.090353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773f8125_c875_4892_9e3a_201595bac469.slice/crio-d6ebe20bac76f9cc3b74430e897326e086b446cc498d0e2a11beec8aa330214a WatchSource:0}: Error finding container d6ebe20bac76f9cc3b74430e897326e086b446cc498d0e2a11beec8aa330214a: Status 404 returned error can't find the container with id d6ebe20bac76f9cc3b74430e897326e086b446cc498d0e2a11beec8aa330214a
Jul 22 01:44:52.092686 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.092096    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eeb6b09_899e_4336_8693_99407d362af3.slice/crio-b59de1e083cb190df420c541f453ec16a020048d4d9a8521f644d4b29ffd2329 WatchSource:0}: Error finding container b59de1e083cb190df420c541f453ec16a020048d4d9a8521f644d4b29ffd2329: Status 404 returned error can't find the container with id b59de1e083cb190df420c541f453ec16a020048d4d9a8521f644d4b29ffd2329
Jul 22 01:44:52.092826 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:44:52.092753    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af1a8f5_ca6a_4a06_8302_5c1065fdf9b7.slice/crio-7a14f9532e665e51cadcd045444b245c5a1e763fd5fe99ba60cd29a226a48d90 WatchSource:0}: Error finding container 7a14f9532e665e51cadcd045444b245c5a1e763fd5fe99ba60cd29a226a48d90: Status 404 returned error can't find the container with id 7a14f9532e665e51cadcd045444b245c5a1e763fd5fe99ba60cd29a226a48d90
Jul 22 01:44:52.096763 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.096368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:52.096763 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.096529    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:44:52.096763 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.096608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:52.096763 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.096706    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:44:52.107643 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.107565    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerStarted","Data":"7a14f9532e665e51cadcd045444b245c5a1e763fd5fe99ba60cd29a226a48d90"}
Jul 22 01:44:52.108839 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.108818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"b59de1e083cb190df420c541f453ec16a020048d4d9a8521f644d4b29ffd2329"}
Jul 22 01:44:52.109764 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.109746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s77x5" event={"ID":"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc","Type":"ContainerStarted","Data":"3f0606cd7fbe20ae3be67752691242f1da5b33865425346d110612a94cdb703d"}
Jul 22 01:44:52.110874 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.110853    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vxpkx" event={"ID":"f7a3c44b-41d2-490e-a053-e6319355233b","Type":"ContainerStarted","Data":"f7062f3980698e3af1f597f8b4ee3ba0e304d945873d990c750828283d87536c"}
Jul 22 01:44:52.112166 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.112141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnfk6" event={"ID":"64c2937f-17fb-4aa5-881e-c859d6f9ae0a","Type":"ContainerStarted","Data":"efced52d96df9ea717ed7ea908ddc2d05a40ef5282682e7b262381360756dfbb"}
Jul 22 01:44:52.113285 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.113264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t9rjn" event={"ID":"773f8125-c875-4892-9e3a-201595bac469","Type":"ContainerStarted","Data":"d6ebe20bac76f9cc3b74430e897326e086b446cc498d0e2a11beec8aa330214a"}
Jul 22 01:44:52.114405 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.114382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n" event={"ID":"c6dff408-2a12-4984-b95b-8937ad3254da","Type":"ContainerStarted","Data":"7390884b9b08de5f1eac528fea8523f45d5a76ee1ca1f8777fd525d88d5878f3"}
Jul 22 01:44:52.115484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.115448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8j9b" event={"ID":"b2a4e8df-2d58-4fa6-a084-484802fabbbf","Type":"ContainerStarted","Data":"686fed6050c7fe40febdbf848f81ac6e059e474841b3fde4e46394b50b5189b1"}
Jul 22 01:44:52.116432 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.116412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4hg5" event={"ID":"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd","Type":"ContainerStarted","Data":"24e0cd8caf892edadf00a3726e0bb70c53a4247a157a58a706c46b28e5adcb19"}
Jul 22 01:44:52.548461 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.547937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:52.548461 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.548178    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:52.548461 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.548219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:52.548461 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.548247    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:44:54.548227477 +0000 UTC m=+6.051085691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:52.548461 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.548306    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:52.548461 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.548357    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:54.548341022 +0000 UTC m=+6.051199239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:52.750358 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.750319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:52.750514 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.750494    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:44:52.750574 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.750516    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:44:52.750574 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.750528    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:52.750684 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:52.750586    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:54.750568261 +0000 UTC m=+6.253426469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:52.963797 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.963672    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 01:39:50 +0000 UTC" deadline="2028-05-05 03:51:45.103368372 +0000 UTC"
Jul 22 01:44:52.963797 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:52.963745    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15674h6m52.139627351s"
Jul 22 01:44:53.100165 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:53.100137    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:53.100347 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:53.100260    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:44:53.132876 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:53.131990    2561 generic.go:356] "Generic (PLEG): container finished" podID="e61819b72a47ad3f811b99f43a91c5fa" containerID="b215589587399d09c7591730360674b3fe57ecd916460429b81bed978c8c43e9" exitCode=0
Jul 22 01:44:53.132876 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:53.132079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" event={"ID":"e61819b72a47ad3f811b99f43a91c5fa","Type":"ContainerDied","Data":"b215589587399d09c7591730360674b3fe57ecd916460429b81bed978c8c43e9"}
Jul 22 01:44:53.144948 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:53.144917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal" event={"ID":"1f6a1ecb72d07fd09a4dca395487a6f2","Type":"ContainerStarted","Data":"03933888a24775796616c759893786d5b2a9203d0be0a07c384a7cbb4b12a643"}
Jul 22 01:44:54.097477 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.096766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:54.097477 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.096907    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:44:54.097477 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.097288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:54.097477 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.097382    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:44:54.160441 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.160403    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" event={"ID":"e61819b72a47ad3f811b99f43a91c5fa","Type":"ContainerStarted","Data":"237ed33a48f03ed733d204f75ac9663cbd5981a3ed7b675d6afcb0e31d4feceb"}
Jul 22 01:44:54.176140 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.176075    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-189.ec2.internal" podStartSLOduration=4.176034209 podStartE2EDuration="4.176034209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:53.170968698 +0000 UTC m=+4.673826928" watchObservedRunningTime="2026-07-22 01:44:54.176034209 +0000 UTC m=+5.678892439"
Jul 22 01:44:54.566069 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.565961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:54.566069 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.566017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:54.566291 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.566164    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:54.566291 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.566224    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:58.566206222 +0000 UTC m=+10.069064449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:54.566873 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.566622    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:54.566873 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.566677    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:44:58.566654559 +0000 UTC m=+10.069512783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:54.768097 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:54.768064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:54.768339 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.768321    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:44:54.768426 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.768349    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:44:54.768426 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.768361    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:54.768426 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:54.768421    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:44:58.768402979 +0000 UTC m=+10.271261201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:55.099357 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:55.098867    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:55.099357 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:55.099023    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:44:56.096286 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:56.096250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:56.096466 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:56.096398    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:44:56.096466 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:56.096433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:56.096570 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:56.096497    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:44:57.096937 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:57.096906    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:57.097384 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:57.097035    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:44:58.096163 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:58.096130    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:58.096350 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:58.096136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:58.096350 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.096290    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:44:58.096459 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.096355    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:44:58.599107 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:58.599067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:58.599574 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:58.599132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:44:58.599574 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.599242    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:58.599574 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.599274    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:58.599574 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.599313    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:45:06.599294241 +0000 UTC m=+18.102152454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:44:58.599574 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.599334    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:06.599323798 +0000 UTC m=+18.102182008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:44:58.800691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:58.800581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:44:58.800867 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.800829    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:44:58.800867 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.800850    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:44:58.800867 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.800861    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:58.801062 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:58.800924    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:06.800905648 +0000 UTC m=+18.303763859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:44:59.097478 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:44:59.097325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:44:59.097478 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:44:59.097451    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:00.097392 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:00.096889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:00.097392 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:00.097010    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:00.097392 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:00.097227    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:00.097392 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:00.097337    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:01.097028 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:01.096518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:01.097028 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:01.096644    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:02.096663 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:02.096629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:02.096663 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:02.096663    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:02.097160 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:02.096787    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:02.097160 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:02.096908    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:03.096669 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:03.096643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:03.097142 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:03.096778    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:04.096820 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:04.096779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:04.097240 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:04.096780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:04.097240 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:04.096935    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:04.097240 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:04.096940    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:05.096413 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:05.096376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:05.096606 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:05.096516    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:06.096210 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:06.096174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:06.096678 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:06.096217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:06.096678 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.096315    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:06.096678 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.096466    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:06.656095 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:06.656060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:06.656373 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:06.656119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:06.656373 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.656224    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:45:06.656373 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.656294    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:45:22.656277783 +0000 UTC m=+34.159135991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:45:06.656373 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.656232    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:45:06.656552 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.656380    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:22.656362135 +0000 UTC m=+34.159220345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:45:06.857933 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:06.857898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:06.858123 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.858055    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:45:06.858123 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.858081    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:45:06.858123 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.858094    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:45:06.858280 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:06.858160    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:22.858139397 +0000 UTC m=+34.360997626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:45:07.096708 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:07.096626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:07.097129 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:07.096757    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:08.096968 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:08.096938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:08.097385 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:08.096979    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:08.097385 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:08.097065    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:08.097385 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:08.097178    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:09.100607 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:09.100526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:09.101083 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:09.100699    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:10.097025 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:10.096988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:10.097218 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:10.096988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:10.097218 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:10.097107    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:10.097218 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:10.097207    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:11.097181 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.096538    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:11.097181 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:11.096867    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:11.191718 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.191676    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s77x5" event={"ID":"cab3a42b-231d-4f4e-bbec-1ed8e9421ecc","Type":"ContainerStarted","Data":"829fd0cbf200f19a3138c849fb559f26cd9154872b08c0125993972c5ea0367f"}
Jul 22 01:45:11.193196 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.193171    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vxpkx" event={"ID":"f7a3c44b-41d2-490e-a053-e6319355233b","Type":"ContainerStarted","Data":"2028def8d6572ce04100b3e69848482a32734e049873bd894e5f72c3f9b46895"}
Jul 22 01:45:11.194785 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.194752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnfk6" event={"ID":"64c2937f-17fb-4aa5-881e-c859d6f9ae0a","Type":"ContainerStarted","Data":"7b250e92ca2db49b3ea69d5c9dbb2c4669d83f3f7f80ec595a5118e665e9d973"}
Jul 22 01:45:11.196366 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.196340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n" event={"ID":"c6dff408-2a12-4984-b95b-8937ad3254da","Type":"ContainerStarted","Data":"81300d76bad9dd9ef7e1a989bc772d69de64804640982f72e63a5788ed4fbdfd"}
Jul 22 01:45:11.197787 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.197724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t8j9b" event={"ID":"b2a4e8df-2d58-4fa6-a084-484802fabbbf","Type":"ContainerStarted","Data":"116cdd271d777448ef069fdad2d65e1aeb76f2632953329b797f70ff50c493a5"}
Jul 22 01:45:11.199048 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.199027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4hg5" event={"ID":"f0f19b6e-6eec-4ad8-af2f-69c5844b7abd","Type":"ContainerStarted","Data":"4faae9de62dcacaa10111de5055f46ed350aed129d78ac396d530d7fa1180d77"}
Jul 22 01:45:11.200450 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.200433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerStarted","Data":"9762bc4a712c7c029416335dfa819937eb342bf5863a6c83a5c26901dba00902"}
Jul 22 01:45:11.208045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.208010    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-189.ec2.internal" podStartSLOduration=21.207979727 podStartE2EDuration="21.207979727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:44:54.176788548 +0000 UTC m=+5.679646777" watchObservedRunningTime="2026-07-22 01:45:11.207979727 +0000 UTC m=+22.710837953"
Jul 22 01:45:11.208405 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.208385    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s77x5" podStartSLOduration=3.432072337 podStartE2EDuration="22.208376928s" totalImagesPullingTime="18.776304591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:11.208117231 +0000 UTC m=+22.710975438" watchObservedRunningTime="2026-07-22 01:45:11.208376928 +0000 UTC m=+22.711235156"
Jul 22 01:45:11.223867 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.223827    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vxpkx" podStartSLOduration=3.417154225 podStartE2EDuration="22.223815089s" totalImagesPullingTime="18.806660864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:11.223572479 +0000 UTC m=+22.726430709" watchObservedRunningTime="2026-07-22 01:45:11.223815089 +0000 UTC m=+22.726673317"
Jul 22 01:45:11.252808 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.252760    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rnfk6" podStartSLOduration=3.443774481 podStartE2EDuration="22.252744747s" totalImagesPullingTime="18.808970266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:11.252284536 +0000 UTC m=+22.755142767" watchObservedRunningTime="2026-07-22 01:45:11.252744747 +0000 UTC m=+22.755602967"
Jul 22 01:45:11.276305 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.276256    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m4hg5" podStartSLOduration=11.849972898 podStartE2EDuration="22.276238382s" totalImagesPullingTime="10.426265484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:11.276136321 +0000 UTC m=+22.778994549" watchObservedRunningTime="2026-07-22 01:45:11.276238382 +0000 UTC m=+22.779096612"
Jul 22 01:45:11.346906 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.346863    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t8j9b" podStartSLOduration=3.566647122 podStartE2EDuration="22.346848655s" totalImagesPullingTime="18.780201533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:11.346394299 +0000 UTC m=+22.849252529" watchObservedRunningTime="2026-07-22 01:45:11.346848655 +0000 UTC m=+22.849706882"
Jul 22 01:45:11.518142 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.518115    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:45:11.518721 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:11.518702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:45:12.096837 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.096653    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:12.096931 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.096707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:12.096931 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:12.096915    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:12.097003 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:12.096981    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:12.204177 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.204105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t9rjn" event={"ID":"773f8125-c875-4892-9e3a-201595bac469","Type":"ContainerStarted","Data":"6267d1568099a77d0b72a57fb577778518e9ca452216385cd0671a84467777f3"}
Jul 22 01:45:12.205484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.205460    2561 generic.go:356] "Generic (PLEG): container finished" podID="3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7" containerID="9762bc4a712c7c029416335dfa819937eb342bf5863a6c83a5c26901dba00902" exitCode=0
Jul 22 01:45:12.205618 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.205541    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerDied","Data":"9762bc4a712c7c029416335dfa819937eb342bf5863a6c83a5c26901dba00902"}
Jul 22 01:45:12.208249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.208230    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:45:12.208641 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.208619    2561 generic.go:356] "Generic (PLEG): container finished" podID="4eeb6b09-899e-4336-8693-99407d362af3" containerID="f72156a5d1f393e8993f316bbd8c399c81f82f3694a5076e943f15443005d26a" exitCode=1
Jul 22 01:45:12.208761 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.208677    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"96b3d826d4fbe4c3e5188451e021fc49ffb1822ff6bb977d864c97b65b60bf14"}
Jul 22 01:45:12.208828 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.208761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"e02d9cf9f76415b8bfb2c8aad91259e4b181d95f396b60a4f3f7c1f813e80cfa"}
Jul 22 01:45:12.208828 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.208813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"55fb594fe93100f55c6f19c097fef9fd1c94e062a3e122133b433baba3742a45"}
Jul 22 01:45:12.209167 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.209143    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:45:12.209264 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.209178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"afbc750fc671aa3da3f2cc7e1599cbaa5facc3d8cc2ee64faa3b933648587dcd"}
Jul 22 01:45:12.209312 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.209264    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerDied","Data":"f72156a5d1f393e8993f316bbd8c399c81f82f3694a5076e943f15443005d26a"}
Jul 22 01:45:12.209344 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.209334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"198686ac1aa1bb63390572fe9bbbca6152d9b51ddba24cdcfd2ec985a9836264"}
Jul 22 01:45:12.209662 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.209649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m4hg5"
Jul 22 01:45:12.222699 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.222661    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t9rjn" podStartSLOduration=4.447576917 podStartE2EDuration="23.222651142s" totalImagesPullingTime="18.775074225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:12.22239097 +0000 UTC m=+23.725249196" watchObservedRunningTime="2026-07-22 01:45:12.222651142 +0000 UTC m=+23.725509371"
Jul 22 01:45:12.383215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:12.383190    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 22 01:45:13.096603 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:13.096479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:13.096603 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:13.096599    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:13.142410 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:13.142124    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T01:45:12.383211447Z","UUID":"61fd1058-a839-49ed-8835-90851930f529","Handler":null,"Name":"","Endpoint":""}
Jul 22 01:45:13.144145 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:13.144124    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 22 01:45:13.144271 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:13.144170    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 22 01:45:13.213138 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:13.213099    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n" event={"ID":"c6dff408-2a12-4984-b95b-8937ad3254da","Type":"ContainerStarted","Data":"32e9d039ef7879bf879035a9dae597b04b7de1819a579194c5d48a7d11ea46f3"}
Jul 22 01:45:14.096323 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:14.096295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:14.096503 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:14.096335    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:14.096503 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:14.096414    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:14.096620 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:14.096561    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:14.216812 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:14.216783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n" event={"ID":"c6dff408-2a12-4984-b95b-8937ad3254da","Type":"ContainerStarted","Data":"7974d2226d5cf9f37ac540e1380fa050bd20e76ae2dce1d9c49fea0cd1c16e66"}
Jul 22 01:45:14.258206 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:14.258163    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gh52n" podStartSLOduration=4.211189125 podStartE2EDuration="25.258142808s" totalImagesPullingTime="21.046953683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:14.257000388 +0000 UTC m=+25.759858616" watchObservedRunningTime="2026-07-22 01:45:14.258142808 +0000 UTC m=+25.761001037"
Jul 22 01:45:15.096234 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:15.096058    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:15.096511 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:15.096322    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:15.222266 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:15.222235    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:45:15.222714 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:15.222688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"1ca31635a436a91aa9521c122ebe0efb4e223c81298a6309b7601103f3a7bf33"}
Jul 22 01:45:16.096864 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:16.096833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:16.096864 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:16.096856    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:16.097137 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:16.096974    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:16.097137 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:16.097106    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:17.096944 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.096699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:17.097557 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:17.096982    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:17.228819 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.228788    2561 generic.go:356] "Generic (PLEG): container finished" podID="3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7" containerID="4882ed9276bdaeb4005db1b060544a6e8916e27f1887dc868b63a9bad7eee675" exitCode=0
Jul 22 01:45:17.228981 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.228875    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerDied","Data":"4882ed9276bdaeb4005db1b060544a6e8916e27f1887dc868b63a9bad7eee675"}
Jul 22 01:45:17.232164 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.232150    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:45:17.232480 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.232463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"e006aff796595d365600a891d015a53f08fbce4c8f59ffb304f8792e700c562f"}
Jul 22 01:45:17.232786 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.232766    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:45:17.232874 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.232833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:45:17.232935 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.232879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:45:17.232982 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.232935    2561 scope.go:117] "RemoveContainer" containerID="f72156a5d1f393e8993f316bbd8c399c81f82f3694a5076e943f15443005d26a"
Jul 22 01:45:17.248810 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.248791    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:45:17.252168 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:17.252148    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:45:18.096350 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.096311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:18.096535 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.096351    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:18.096535 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:18.096457    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:18.096627 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:18.096550    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:18.238446 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.238427    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:45:18.238885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.238857    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" event={"ID":"4eeb6b09-899e-4336-8693-99407d362af3","Type":"ContainerStarted","Data":"85ad3907ea394aec8b96aefda37f79042d46e43b63d1ec654c12dfc257c76797"}
Jul 22 01:45:18.269828 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.269772    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z" podStartSLOduration=9.930347675 podStartE2EDuration="29.269752023s" totalImagesPullingTime="19.339404348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:18.268131677 +0000 UTC m=+29.770989942" watchObservedRunningTime="2026-07-22 01:45:18.269752023 +0000 UTC m=+29.772610253"
Jul 22 01:45:18.535280 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.535248    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sqzb4"]
Jul 22 01:45:18.535411 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.535370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:18.535476 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:18.535443    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:18.538514 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.538490    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh2rd"]
Jul 22 01:45:18.538617 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.538585    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:18.538683 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:18.538665    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:18.548472 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.548449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wpwxr"]
Jul 22 01:45:18.548582 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:18.548527    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:18.548649 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:18.548598    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:19.243120 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:19.243089    2561 generic.go:356] "Generic (PLEG): container finished" podID="3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7" containerID="a086e180038a456125686084588c2c4ab2fcd67ef316cbf69f5b859ab66085cd" exitCode=0
Jul 22 01:45:19.243489 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:19.243179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerDied","Data":"a086e180038a456125686084588c2c4ab2fcd67ef316cbf69f5b859ab66085cd"}
Jul 22 01:45:20.096750 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:20.096696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:20.096750 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:20.096753    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:20.096983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:20.096697    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:20.096983 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:20.096849    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:20.096983 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:20.096947    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:20.097125 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:20.097017    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:21.250046 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:21.250015    2561 generic.go:356] "Generic (PLEG): container finished" podID="3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7" containerID="ff764f48d1885b79f05b0d9f42757076d39d75bcda8c3d38fb4320cdd2eeebef" exitCode=0
Jul 22 01:45:21.250606 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:21.250060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerDied","Data":"ff764f48d1885b79f05b0d9f42757076d39d75bcda8c3d38fb4320cdd2eeebef"}
Jul 22 01:45:22.096889 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.096859    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:22.097068 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.096862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:22.097068 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.096973    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:22.097185 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.097079    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:22.097185 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.096862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:22.097277 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.097198    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:22.322207 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.322167    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:22.420525 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.420449    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:22.521225 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.521192    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:22.620323 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.620288    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:22.675212 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.675136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:22.675212 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.675182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:22.675408 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.675306    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:45:22.675408 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.675353    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs podName:6e53db84-b279-44fb-a467-07f335547a01 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:54.675339876 +0000 UTC m=+66.178198086 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs") pod "network-metrics-daemon-rh2rd" (UID: "6e53db84-b279-44fb-a467-07f335547a01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 01:45:22.675408 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.675307    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 01:45:22.675532 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.675440    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret podName:e297bc0c-22b3-4e81-a4b4-ae416d6ae24e nodeName:}" failed. No retries permitted until 2026-07-22 01:45:54.675424577 +0000 UTC m=+66.178282800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret") pod "global-pull-secret-syncer-sqzb4" (UID: "e297bc0c-22b3-4e81-a4b4-ae416d6ae24e") : object "kube-system"/"original-pull-secret" not registered
Jul 22 01:45:22.720964 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.720933    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:22.821579 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.821545    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:22.876606 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.876573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:22.876798 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.876767    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 01:45:22.876798 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.876792    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 01:45:22.876905 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.876806    2561 projected.go:196] Error preparing data for projected volume kube-api-access-25xws for pod openshift-network-diagnostics/network-check-target-wpwxr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:45:22.876905 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:22.876870    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws podName:be905c4d-e0e9-4a07-83fc-7c9063b709d6 nodeName:}" failed. No retries permitted until 2026-07-22 01:45:54.876849089 +0000 UTC m=+66.379707311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-25xws" (UniqueName: "kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws") pod "network-check-target-wpwxr" (UID: "be905c4d-e0e9-4a07-83fc-7c9063b709d6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 01:45:22.922867 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:22.922829    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:23.398963 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:23.398924    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:23.457744 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:23.457700    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:23.806230 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:23.806151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:23.872952 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:23.872916    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:24.096671 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.096590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:24.096852 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.096590    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:24.096852 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.096719    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sqzb4" podUID="e297bc0c-22b3-4e81-a4b4-ae416d6ae24e"
Jul 22 01:45:24.096852 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.096751    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:24.096852 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.096824    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wpwxr" podUID="be905c4d-e0e9-4a07-83fc-7c9063b709d6"
Jul 22 01:45:24.097058 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.096912    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh2rd" podUID="6e53db84-b279-44fb-a467-07f335547a01"
Jul 22 01:45:24.727541 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.727464    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-189.ec2.internal" event="NodeReady"
Jul 22 01:45:24.728017 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.727616    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 22 01:45:24.763744 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.763699    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f95749bb-srd25"]
Jul 22 01:45:24.768597 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.768577    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.771411 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.771392    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-86kzc\""
Jul 22 01:45:24.772144 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.772122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 22 01:45:24.772400 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.772336    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 22 01:45:24.772532 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.772337    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 22 01:45:24.774300 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.774279    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jwnlg"]
Jul 22 01:45:24.777918 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.777890    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z74wb"]
Jul 22 01:45:24.778022 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.777920    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 22 01:45:24.778077 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.778049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.781632 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.780281    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 22 01:45:24.782445 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.782424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 22 01:45:24.782666 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.782636    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nz6sc\""
Jul 22 01:45:24.782798 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.782774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:24.785426 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.785407    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 22 01:45:24.785655 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.785634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 22 01:45:24.785758 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.785699    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 22 01:45:24.785758 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.785709    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xrls9\""
Jul 22 01:45:24.788027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.787988    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f95749bb-srd25"]
Jul 22 01:45:24.789388 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.789370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jwnlg"]
Jul 22 01:45:24.789664 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.789645    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z74wb"]
Jul 22 01:45:24.891886 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.891848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vgc\" (UniqueName: \"kubernetes.io/projected/30f6828a-7509-4862-b361-7946baf84f45-kube-api-access-22vgc\") pod \"ingress-canary-z74wb\" (UID: \"30f6828a-7509-4862-b361-7946baf84f45\") " pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:24.892067 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.891899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892067 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.891925    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ffa62a-77ee-4485-b3ac-7e0872408dee-ca-trust-extracted\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892067 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-certificates\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892067 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-installation-pull-secrets\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfb5\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-kube-api-access-6lfb5\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52e130bf-16ac-457b-bff4-e49e11b0e3bb-config-volume\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f6828a-7509-4862-b361-7946baf84f45-cert\") pod \"ingress-canary-z74wb\" (UID: \"30f6828a-7509-4862-b361-7946baf84f45\") " pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-image-registry-private-configuration\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-bound-sa-token\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-trusted-ca\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.892551 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e130bf-16ac-457b-bff4-e49e11b0e3bb-tmp-dir\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.892551 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.892355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztt2\" (UniqueName: \"kubernetes.io/projected/52e130bf-16ac-457b-bff4-e49e11b0e3bb-kube-api-access-fztt2\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.993330 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993245    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vgc\" (UniqueName: \"kubernetes.io/projected/30f6828a-7509-4862-b361-7946baf84f45-kube-api-access-22vgc\") pod \"ingress-canary-z74wb\" (UID: \"30f6828a-7509-4862-b361-7946baf84f45\") " pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:24.993330 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ffa62a-77ee-4485-b3ac-7e0872408dee-ca-trust-extracted\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.993394    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.993408    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f95749bb-srd25: secret "image-registry-tls" not found
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-certificates\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.993465    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls podName:57ffa62a-77ee-4485-b3ac-7e0872408dee nodeName:}" failed. No retries permitted until 2026-07-22 01:45:25.493445699 +0000 UTC m=+36.996303922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls") pod "image-registry-7f95749bb-srd25" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee") : secret "image-registry-tls" not found
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-installation-pull-secrets\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfb5\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-kube-api-access-6lfb5\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52e130bf-16ac-457b-bff4-e49e11b0e3bb-config-volume\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f6828a-7509-4862-b361-7946baf84f45-cert\") pod \"ingress-canary-z74wb\" (UID: \"30f6828a-7509-4862-b361-7946baf84f45\") " pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-image-registry-private-configuration\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-bound-sa-token\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-trusted-ca\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.993812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ffa62a-77ee-4485-b3ac-7e0872408dee-ca-trust-extracted\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.993983 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.993963    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:45:24.994306 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:24.994025    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls podName:52e130bf-16ac-457b-bff4-e49e11b0e3bb nodeName:}" failed. No retries permitted until 2026-07-22 01:45:25.494010668 +0000 UTC m=+36.996868879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls") pod "dns-default-jwnlg" (UID: "52e130bf-16ac-457b-bff4-e49e11b0e3bb") : secret "dns-default-metrics-tls" not found
Jul 22 01:45:24.994306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.994052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e130bf-16ac-457b-bff4-e49e11b0e3bb-tmp-dir\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.994306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.994068    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-certificates\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.994306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.994087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztt2\" (UniqueName: \"kubernetes.io/projected/52e130bf-16ac-457b-bff4-e49e11b0e3bb-kube-api-access-fztt2\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.994306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.994174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52e130bf-16ac-457b-bff4-e49e11b0e3bb-config-volume\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.994306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.994296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e130bf-16ac-457b-bff4-e49e11b0e3bb-tmp-dir\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:24.994757 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.994718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-trusted-ca\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.998395 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.998371    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-installation-pull-secrets\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.998516 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.998398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-image-registry-private-configuration\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:24.998516 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:24.998434    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30f6828a-7509-4862-b361-7946baf84f45-cert\") pod \"ingress-canary-z74wb\" (UID: \"30f6828a-7509-4862-b361-7946baf84f45\") " pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:25.004702 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.004677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vgc\" (UniqueName: \"kubernetes.io/projected/30f6828a-7509-4862-b361-7946baf84f45-kube-api-access-22vgc\") pod \"ingress-canary-z74wb\" (UID: \"30f6828a-7509-4862-b361-7946baf84f45\") " pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:25.004854 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.004833    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-bound-sa-token\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:25.005111 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.005089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztt2\" (UniqueName: \"kubernetes.io/projected/52e130bf-16ac-457b-bff4-e49e11b0e3bb-kube-api-access-fztt2\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:25.005867 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.005844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfb5\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-kube-api-access-6lfb5\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:25.099351 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.099325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z74wb"
Jul 22 01:45:25.233211 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.233043    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z74wb"]
Jul 22 01:45:25.236158 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:25.236129    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f6828a_7509_4862_b361_7946baf84f45.slice/crio-b02a6f689b700b06dea02d9055cb01c913767526430fd899c0813d855a463bd5 WatchSource:0}: Error finding container b02a6f689b700b06dea02d9055cb01c913767526430fd899c0813d855a463bd5: Status 404 returned error can't find the container with id b02a6f689b700b06dea02d9055cb01c913767526430fd899c0813d855a463bd5
Jul 22 01:45:25.261259 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.261231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z74wb" event={"ID":"30f6828a-7509-4862-b361-7946baf84f45","Type":"ContainerStarted","Data":"b02a6f689b700b06dea02d9055cb01c913767526430fd899c0813d855a463bd5"}
Jul 22 01:45:25.263802 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.263778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerStarted","Data":"34fff64d189c726dc0a66da7782391adc7458b419cd983b1c5ff14a01cb13d1d"}
Jul 22 01:45:25.498392 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.498314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:25.498392 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:25.498373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:25.498557 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:25.498465    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:45:25.498557 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:25.498493    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:45:25.498557 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:25.498503    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f95749bb-srd25: secret "image-registry-tls" not found
Jul 22 01:45:25.498557 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:25.498527    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls podName:52e130bf-16ac-457b-bff4-e49e11b0e3bb nodeName:}" failed. No retries permitted until 2026-07-22 01:45:26.49850946 +0000 UTC m=+38.001367666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls") pod "dns-default-jwnlg" (UID: "52e130bf-16ac-457b-bff4-e49e11b0e3bb") : secret "dns-default-metrics-tls" not found
Jul 22 01:45:25.498557 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:25.498544    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls podName:57ffa62a-77ee-4485-b3ac-7e0872408dee nodeName:}" failed. No retries permitted until 2026-07-22 01:45:26.498537591 +0000 UTC m=+38.001395798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls") pod "image-registry-7f95749bb-srd25" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee") : secret "image-registry-tls" not found
Jul 22 01:45:26.096955 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.096922    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:26.096955 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.096945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:26.097655 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.096976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:26.100045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.100019    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 01:45:26.101353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.101134    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 01:45:26.101353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.101204    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 01:45:26.101353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.101145    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 01:45:26.101353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.101234    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9wgwz\""
Jul 22 01:45:26.101353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.101213    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cgfms\""
Jul 22 01:45:26.269477 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.269383    2561 generic.go:356] "Generic (PLEG): container finished" podID="3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7" containerID="34fff64d189c726dc0a66da7782391adc7458b419cd983b1c5ff14a01cb13d1d" exitCode=0
Jul 22 01:45:26.269477 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.269434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerDied","Data":"34fff64d189c726dc0a66da7782391adc7458b419cd983b1c5ff14a01cb13d1d"}
Jul 22 01:45:26.506151 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.506124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:26.506446 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:26.506136    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:45:26.506446 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.506246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:26.506446 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:26.506289    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls podName:52e130bf-16ac-457b-bff4-e49e11b0e3bb nodeName:}" failed. No retries permitted until 2026-07-22 01:45:28.506265687 +0000 UTC m=+40.009123909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls") pod "dns-default-jwnlg" (UID: "52e130bf-16ac-457b-bff4-e49e11b0e3bb") : secret "dns-default-metrics-tls" not found
Jul 22 01:45:26.506446 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:26.506352    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:45:26.506446 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:26.506363    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f95749bb-srd25: secret "image-registry-tls" not found
Jul 22 01:45:26.506446 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:26.506412    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls podName:57ffa62a-77ee-4485-b3ac-7e0872408dee nodeName:}" failed. No retries permitted until 2026-07-22 01:45:28.506397587 +0000 UTC m=+40.009255797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls") pod "image-registry-7f95749bb-srd25" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee") : secret "image-registry-tls" not found
Jul 22 01:45:26.770237 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.770204    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"]
Jul 22 01:45:26.774716 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.774691    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"]
Jul 22 01:45:26.774855 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.774838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:26.777280 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.777259    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 22 01:45:26.777365 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.777321    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 22 01:45:26.777365 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.777334    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jc2wx\""
Jul 22 01:45:26.777452 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.777264    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 22 01:45:26.777745 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.777700    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"]
Jul 22 01:45:26.777812 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.777802    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:26.778291 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.778275    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 22 01:45:26.780068 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.780041    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 22 01:45:26.780197 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.780118    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 22 01:45:26.780306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.780278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 22 01:45:26.780470 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.780448    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 22 01:45:26.780672 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.780649    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:26.783413 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.783243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 22 01:45:26.783413 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.783357    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"]
Jul 22 01:45:26.787335 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.787007    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"]
Jul 22 01:45:26.788436 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.788412    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"]
Jul 22 01:45:26.909369 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-ca\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:26.909537 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:26.909537 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909486    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8342a5-09b9-46e5-87e0-2b042a9bb940-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:26.909537 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:26.909702 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909608    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhr8t\" (UniqueName: \"kubernetes.io/projected/8229152d-685f-4291-8547-900859dcc016-kube-api-access-bhr8t\") pod \"managed-serviceaccount-addon-agent-54686bcb56-djrsb\" (UID: \"8229152d-685f-4291-8547-900859dcc016\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:26.909702 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e453fd-0074-477a-941b-2464b74e3d12-tmp\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:26.909803 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8229152d-685f-4291-8547-900859dcc016-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54686bcb56-djrsb\" (UID: \"8229152d-685f-4291-8547-900859dcc016\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:26.909849 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b0e453fd-0074-477a-941b-2464b74e3d12-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:26.909887 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcw9b\" (UniqueName: \"kubernetes.io/projected/b0e453fd-0074-477a-941b-2464b74e3d12-kube-api-access-gcw9b\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:26.909931 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9j9\" (UniqueName: \"kubernetes.io/projected/5f8342a5-09b9-46e5-87e0-2b042a9bb940-kube-api-access-vv9j9\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:26.909931 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:26.909924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-hub\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.010947 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.010909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhr8t\" (UniqueName: \"kubernetes.io/projected/8229152d-685f-4291-8547-900859dcc016-kube-api-access-bhr8t\") pod \"managed-serviceaccount-addon-agent-54686bcb56-djrsb\" (UID: \"8229152d-685f-4291-8547-900859dcc016\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:27.011123 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.010959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e453fd-0074-477a-941b-2464b74e3d12-tmp\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.011123 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8229152d-685f-4291-8547-900859dcc016-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54686bcb56-djrsb\" (UID: \"8229152d-685f-4291-8547-900859dcc016\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:27.011123 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b0e453fd-0074-477a-941b-2464b74e3d12-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.011123 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcw9b\" (UniqueName: \"kubernetes.io/projected/b0e453fd-0074-477a-941b-2464b74e3d12-kube-api-access-gcw9b\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.011123 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9j9\" (UniqueName: \"kubernetes.io/projected/5f8342a5-09b9-46e5-87e0-2b042a9bb940-kube-api-access-vv9j9\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.011399 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011134    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-hub\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.011473 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-ca\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.011473 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e453fd-0074-477a-941b-2464b74e3d12-tmp\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.011664 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.011664 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011532    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8342a5-09b9-46e5-87e0-2b042a9bb940-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.011664 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.011555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.012283 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.012243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8342a5-09b9-46e5-87e0-2b042a9bb940-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.015295 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.015206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.015396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.015304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8229152d-685f-4291-8547-900859dcc016-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54686bcb56-djrsb\" (UID: \"8229152d-685f-4291-8547-900859dcc016\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:27.015396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.015312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-ca\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.015396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.015302    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-hub\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.015958 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.015935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b0e453fd-0074-477a-941b-2464b74e3d12-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.016764 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.016748    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5f8342a5-09b9-46e5-87e0-2b042a9bb940-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.020715 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.020661    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9j9\" (UniqueName: \"kubernetes.io/projected/5f8342a5-09b9-46e5-87e0-2b042a9bb940-kube-api-access-vv9j9\") pod \"cluster-proxy-proxy-agent-7b86c9c78c-gsbmh\" (UID: \"5f8342a5-09b9-46e5-87e0-2b042a9bb940\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.020886 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.020863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhr8t\" (UniqueName: \"kubernetes.io/projected/8229152d-685f-4291-8547-900859dcc016-kube-api-access-bhr8t\") pod \"managed-serviceaccount-addon-agent-54686bcb56-djrsb\" (UID: \"8229152d-685f-4291-8547-900859dcc016\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:27.021107 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.021090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcw9b\" (UniqueName: \"kubernetes.io/projected/b0e453fd-0074-477a-941b-2464b74e3d12-kube-api-access-gcw9b\") pod \"klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw\" (UID: \"b0e453fd-0074-477a-941b-2464b74e3d12\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.094130 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.094088    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"
Jul 22 01:45:27.101029 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.100999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:27.108691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.108666    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:27.275904 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.275817    2561 generic.go:356] "Generic (PLEG): container finished" podID="3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7" containerID="e2c91261fa70d13bf95d3ed8f774b56039bcb9a3258347303b53b2ae15bb5a93" exitCode=0
Jul 22 01:45:27.276059 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.275910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerDied","Data":"e2c91261fa70d13bf95d3ed8f774b56039bcb9a3258347303b53b2ae15bb5a93"}
Jul 22 01:45:27.542990 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.542760    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"]
Jul 22 01:45:27.544958 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:27.544937    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e453fd_0074_477a_941b_2464b74e3d12.slice/crio-0e0c424ba2f1e548ebfc9756bd6f651fd08faf44758f1e2aa51b297d6fe73b57 WatchSource:0}: Error finding container 0e0c424ba2f1e548ebfc9756bd6f651fd08faf44758f1e2aa51b297d6fe73b57: Status 404 returned error can't find the container with id 0e0c424ba2f1e548ebfc9756bd6f651fd08faf44758f1e2aa51b297d6fe73b57
Jul 22 01:45:27.548902 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.548876    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"]
Jul 22 01:45:27.549776 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:27.549753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb"]
Jul 22 01:45:27.550606 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:27.550513    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8229152d_685f_4291_8547_900859dcc016.slice/crio-c1050a384c2f1a52bbfe3c1ad7c23a85c55347ea0bd2342c04223756afa8e870 WatchSource:0}: Error finding container c1050a384c2f1a52bbfe3c1ad7c23a85c55347ea0bd2342c04223756afa8e870: Status 404 returned error can't find the container with id c1050a384c2f1a52bbfe3c1ad7c23a85c55347ea0bd2342c04223756afa8e870
Jul 22 01:45:27.552274 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:27.552253    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f8342a5_09b9_46e5_87e0_2b042a9bb940.slice/crio-f260e1ecc6112a40198e8a2a00b4f72b5a29e474bb165c588b1f1042c5b248f6 WatchSource:0}: Error finding container f260e1ecc6112a40198e8a2a00b4f72b5a29e474bb165c588b1f1042c5b248f6: Status 404 returned error can't find the container with id f260e1ecc6112a40198e8a2a00b4f72b5a29e474bb165c588b1f1042c5b248f6
Jul 22 01:45:28.282187 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.282153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" event={"ID":"3af1a8f5-ca6a-4a06-8302-5c1065fdf9b7","Type":"ContainerStarted","Data":"3dbfe2f317054073297ff1a8185505464af3c33dedc5654d2bae658b1c274ec1"}
Jul 22 01:45:28.283428 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.283409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z74wb" event={"ID":"30f6828a-7509-4862-b361-7946baf84f45","Type":"ContainerStarted","Data":"36d6d819a58b18de624430c152542e1b992cff02dabb0a7110041985f0b891e2"}
Jul 22 01:45:28.284303 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.284284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw" event={"ID":"b0e453fd-0074-477a-941b-2464b74e3d12","Type":"ContainerStarted","Data":"0e0c424ba2f1e548ebfc9756bd6f651fd08faf44758f1e2aa51b297d6fe73b57"}
Jul 22 01:45:28.285134 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.285113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" event={"ID":"5f8342a5-09b9-46e5-87e0-2b042a9bb940","Type":"ContainerStarted","Data":"f260e1ecc6112a40198e8a2a00b4f72b5a29e474bb165c588b1f1042c5b248f6"}
Jul 22 01:45:28.285961 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.285945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb" event={"ID":"8229152d-685f-4291-8547-900859dcc016","Type":"ContainerStarted","Data":"c1050a384c2f1a52bbfe3c1ad7c23a85c55347ea0bd2342c04223756afa8e870"}
Jul 22 01:45:28.308997 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.308960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v8zbk" podStartSLOduration=10.185254612 podStartE2EDuration="39.308947653s" totalImagesPullingTime="29.123693041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:28.307707099 +0000 UTC m=+39.810565349" watchObservedRunningTime="2026-07-22 01:45:28.308947653 +0000 UTC m=+39.811805923"
Jul 22 01:45:28.329436 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.329392    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z74wb" podStartSLOduration=2.140583338 podStartE2EDuration="4.329374714s" totalImagesPullingTime="2.188791376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:28.329108704 +0000 UTC m=+39.831966935" watchObservedRunningTime="2026-07-22 01:45:28.329374714 +0000 UTC m=+39.832232946"
Jul 22 01:45:28.449169 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.449066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:28.512831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.512794    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:28.525288 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.525257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:28.525456 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.525333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:28.525456 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:28.525414    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:45:28.525456 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:28.525433    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f95749bb-srd25: secret "image-registry-tls" not found
Jul 22 01:45:28.525601 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:28.525464    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:45:28.525601 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:28.525502    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls podName:57ffa62a-77ee-4485-b3ac-7e0872408dee nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.525482698 +0000 UTC m=+44.028340929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls") pod "image-registry-7f95749bb-srd25" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee") : secret "image-registry-tls" not found
Jul 22 01:45:28.525601 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:28.525519    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls podName:52e130bf-16ac-457b-bff4-e49e11b0e3bb nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.525510998 +0000 UTC m=+44.028369206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls") pod "dns-default-jwnlg" (UID: "52e130bf-16ac-457b-bff4-e49e11b0e3bb") : secret "dns-default-metrics-tls" not found
Jul 22 01:45:28.574554 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.574518    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:28.738677 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.738578    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:28.838524 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.838485    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:28.938947 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:28.938908    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.030682 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.030649    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.039765 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.039594    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d7zvh"]
Jul 22 01:45:29.055601 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.055571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7zvh"]
Jul 22 01:45:29.055775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.055691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.058978 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.058322    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 22 01:45:29.058978 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.058330    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 22 01:45:29.058978 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.058583    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 22 01:45:29.058978 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.058843    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ldh2d\""
Jul 22 01:45:29.059755 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.059557    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 22 01:45:29.093182 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.093154    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.231148 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.231114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32a06c56-9fab-49ba-88ce-6606585a4e0f-crio-socket\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.231148 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.231160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.231148 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.231233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfqg\" (UniqueName: \"kubernetes.io/projected/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-api-access-5vfqg\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.231148 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.231265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32a06c56-9fab-49ba-88ce-6606585a4e0f-data-volume\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.231148 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.231289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.332078 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.332000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfqg\" (UniqueName: \"kubernetes.io/projected/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-api-access-5vfqg\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.332078 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.332050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32a06c56-9fab-49ba-88ce-6606585a4e0f-data-volume\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.332078 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.332075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.332893 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.332167    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32a06c56-9fab-49ba-88ce-6606585a4e0f-crio-socket\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.332893 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.332190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.332893 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.332347    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:29.332893 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.332409    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:29.832387947 +0000 UTC m=+41.335246173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:29.333560 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.333539    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:45:29.333651 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.333612    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:29.833593909 +0000 UTC m=+41.336452133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : configmap "kube-rbac-proxy" not found
Jul 22 01:45:29.333708 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.333649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32a06c56-9fab-49ba-88ce-6606585a4e0f-crio-socket\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.333708 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.333231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32a06c56-9fab-49ba-88ce-6606585a4e0f-data-volume\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.348304 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.347552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfqg\" (UniqueName: \"kubernetes.io/projected/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-api-access-5vfqg\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.364934 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.364907    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.475722 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.475687    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.713168 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.713060    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.812677 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.812643    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:29.839623 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.839310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.839623 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:29.839419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:29.839623 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.839506    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:45:29.839623 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.839549    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:29.839623 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.839565    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:30.839548373 +0000 UTC m=+42.342406580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : configmap "kube-rbac-proxy" not found
Jul 22 01:45:29.839623 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:29.839597    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:30.839581919 +0000 UTC m=+42.342440128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:30.617276 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:30.617245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8j9b_b2a4e8df-2d58-4fa6-a084-484802fabbbf/dns-node-resolver/0.log"
Jul 22 01:45:30.847673 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:30.847626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:30.847846 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:30.847755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:30.847846 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:30.847783    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:45:30.847948 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:30.847853    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.847833023 +0000 UTC m=+44.350691252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : configmap "kube-rbac-proxy" not found
Jul 22 01:45:30.847948 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:30.847870    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:30.847948 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:30.847918    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:32.847903115 +0000 UTC m=+44.350761324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:31.281306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:31.281267    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:31.334894 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:31.334865    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:31.618109 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:31.618026    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s77x5_cab3a42b-231d-4f4e-bbec-1ed8e9421ecc/node-ca/0.log"
Jul 22 01:45:32.182994 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:32.182965    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:32.417916 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:32.417882    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z74wb_30f6828a-7509-4862-b361-7946baf84f45/serve-healthcheck-canary/0.log"
Jul 22 01:45:32.565564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:32.565533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:32.565758 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:32.565666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:32.565758 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.565724    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 01:45:32.565863 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.565781    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 22 01:45:32.565863 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.565793    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f95749bb-srd25: secret "image-registry-tls" not found
Jul 22 01:45:32.565863 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.565813    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls podName:52e130bf-16ac-457b-bff4-e49e11b0e3bb nodeName:}" failed. No retries permitted until 2026-07-22 01:45:40.565791568 +0000 UTC m=+52.068649781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls") pod "dns-default-jwnlg" (UID: "52e130bf-16ac-457b-bff4-e49e11b0e3bb") : secret "dns-default-metrics-tls" not found
Jul 22 01:45:32.565863 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.565831    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls podName:57ffa62a-77ee-4485-b3ac-7e0872408dee nodeName:}" failed. No retries permitted until 2026-07-22 01:45:40.565820109 +0000 UTC m=+52.068678317 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls") pod "image-registry-7f95749bb-srd25" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee") : secret "image-registry-tls" not found
Jul 22 01:45:32.869509 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:32.869431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:32.869939 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:32.869535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:32.869939 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.869599    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:32.869939 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.869657    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 01:45:32.869939 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.869674    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:36.869658766 +0000 UTC m=+48.372516974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:32.869939 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:32.869703    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:36.869687524 +0000 UTC m=+48.372545734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : configmap "kube-rbac-proxy" not found
Jul 22 01:45:33.572117 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:33.572086    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:33.674005 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:33.673959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:33.774956 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:33.774920    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:33.833100 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:33.833066    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:33.890613 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:33.890577    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:34.001078 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:34.001039    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:34.102765 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:34.102634    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:34.409463 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:34.409381    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:34.668582 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:34.668498    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:35.308104 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:35.308069    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" event={"ID":"5f8342a5-09b9-46e5-87e0-2b042a9bb940","Type":"ContainerStarted","Data":"d1b636b2c10ea7a64b06a5d552fb40f9dece39d7fa7932ea005d7c20c42d9caf"}
Jul 22 01:45:35.309643 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:35.309613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb" event={"ID":"8229152d-685f-4291-8547-900859dcc016","Type":"ContainerStarted","Data":"1186dbba060ae6038b40bebe2a133efd7005312539053aec30d5e74d136ccd82"}
Jul 22 01:45:35.474723 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:35.474699    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:35.579972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:35.579945    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:35.682163 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:35.682097    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:35.869235 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:35.869206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:36.075668 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.075628    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:36.277238 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.277206    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:36.314243 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.314214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw" event={"ID":"b0e453fd-0074-477a-941b-2464b74e3d12","Type":"ContainerStarted","Data":"2a737afbfb57817139ae78825b7732d90f836538ceedef3caefaeb726d44bc84"}
Jul 22 01:45:36.314642 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.314413    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:36.316249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.316225    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw"
Jul 22 01:45:36.316598 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.316576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" event={"ID":"5f8342a5-09b9-46e5-87e0-2b042a9bb940","Type":"ContainerStarted","Data":"e76a706b4d254beb158f3aaacfbb948264af7b24a845057951aff145e03581b9"}
Jul 22 01:45:36.316687 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.316610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" event={"ID":"5f8342a5-09b9-46e5-87e0-2b042a9bb940","Type":"ContainerStarted","Data":"2041ebd1357e05a127204026ca4c94c44883a622131afc30d3963f1627489d72"}
Jul 22 01:45:36.333224 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.333149    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bf5dcb5d7-rqvtw" podStartSLOduration=2.6616567829999997 podStartE2EDuration="10.333136426s" totalImagesPullingTime="7.671479643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:36.332259417 +0000 UTC m=+47.835117648" watchObservedRunningTime="2026-07-22 01:45:36.333136426 +0000 UTC m=+47.835994818"
Jul 22 01:45:36.358850 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.358822    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:36.369902 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.369864    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54686bcb56-djrsb" podStartSLOduration=2.745624036 podStartE2EDuration="10.369851134s" totalImagesPullingTime="7.624227098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:36.369123797 +0000 UTC m=+47.871982024" watchObservedRunningTime="2026-07-22 01:45:36.369851134 +0000 UTC m=+47.872709362"
Jul 22 01:45:36.472771 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.472723    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:36.681028 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.680934    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:36.901934 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.901900    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:36.902106 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.901982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:36.902106 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:36.902094    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:36.902224 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:45:36.902158    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls podName:32a06c56-9fab-49ba-88ce-6606585a4e0f nodeName:}" failed. No retries permitted until 2026-07-22 01:45:44.902143347 +0000 UTC m=+56.405001554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d7zvh" (UID: "32a06c56-9fab-49ba-88ce-6606585a4e0f") : secret "insights-runtime-extractor-tls" not found
Jul 22 01:45:36.902552 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:36.902522    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32a06c56-9fab-49ba-88ce-6606585a4e0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:37.103945 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:37.103900    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" podUID="5f8342a5-09b9-46e5-87e0-2b042a9bb940" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 22 01:45:37.266655 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:37.266625    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:37.595875 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:37.595845    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:37.867108 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:37.867038    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:38.071168 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:38.071139    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:38.267854 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:38.267823    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:38.468106 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:38.468068    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:38.569324 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:38.569244    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:38.668815 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:38.668787    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:39.865176 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:39.865119    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:40.075989 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.075958    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:40.264252 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.264219    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:40.465450 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.465420    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:40.628386 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.628318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:40.628386 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.628359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:40.630667 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.630634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52e130bf-16ac-457b-bff4-e49e11b0e3bb-metrics-tls\") pod \"dns-default-jwnlg\" (UID: \"52e130bf-16ac-457b-bff4-e49e11b0e3bb\") " pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:40.630857 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.630838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"image-registry-7f95749bb-srd25\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") " pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:40.662891 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.662863    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:40.679853 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.679834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:40.693972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.693952    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:40.826805 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.826758    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" podStartSLOduration=7.178682553 podStartE2EDuration="14.826724117s" totalImagesPullingTime="7.648041564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:36.389482354 +0000 UTC m=+47.892340583" watchObservedRunningTime="2026-07-22 01:45:40.826724117 +0000 UTC m=+52.329582340"
Jul 22 01:45:40.827455 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.827436    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f95749bb-srd25"]
Jul 22 01:45:40.829997 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:40.829965    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ffa62a_77ee_4485_b3ac_7e0872408dee.slice/crio-262a0756c409b4db665f9f56ae18e1c12c7ece0fc9dc591ca01bb865bef17778 WatchSource:0}: Error finding container 262a0756c409b4db665f9f56ae18e1c12c7ece0fc9dc591ca01bb865bef17778: Status 404 returned error can't find the container with id 262a0756c409b4db665f9f56ae18e1c12c7ece0fc9dc591ca01bb865bef17778
Jul 22 01:45:40.845109 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.845089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jwnlg"]
Jul 22 01:45:40.847810 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:40.847784    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e130bf_16ac_457b_bff4_e49e11b0e3bb.slice/crio-15dd8cec9b305082ecdd1d612d1fec1fd129669ef35a90c70b10c355d0937037 WatchSource:0}: Error finding container 15dd8cec9b305082ecdd1d612d1fec1fd129669ef35a90c70b10c355d0937037: Status 404 returned error can't find the container with id 15dd8cec9b305082ecdd1d612d1fec1fd129669ef35a90c70b10c355d0937037
Jul 22 01:45:40.911127 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:40.911103    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:41.332817 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:41.332772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jwnlg" event={"ID":"52e130bf-16ac-457b-bff4-e49e11b0e3bb","Type":"ContainerStarted","Data":"15dd8cec9b305082ecdd1d612d1fec1fd129669ef35a90c70b10c355d0937037"}
Jul 22 01:45:41.334165 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:41.334141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f95749bb-srd25" event={"ID":"57ffa62a-77ee-4485-b3ac-7e0872408dee","Type":"ContainerStarted","Data":"db35b7cd353e906649339fb97d4607da45aca3092debf9eb40a7f621eaf78cfc"}
Jul 22 01:45:41.334292 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:41.334178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f95749bb-srd25" event={"ID":"57ffa62a-77ee-4485-b3ac-7e0872408dee","Type":"ContainerStarted","Data":"262a0756c409b4db665f9f56ae18e1c12c7ece0fc9dc591ca01bb865bef17778"}
Jul 22 01:45:41.334292 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:41.334281    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:45:41.353038 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:41.352990    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f95749bb-srd25" podStartSLOduration=33.352976885 podStartE2EDuration="33.352976885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:41.351973304 +0000 UTC m=+52.854831572" watchObservedRunningTime="2026-07-22 01:45:41.352976885 +0000 UTC m=+52.855835104"
Jul 22 01:45:41.868905 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:41.868864    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:42.068673 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:42.068638    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:42.279012 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:42.278985    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:42.480179 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:42.480151    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:42.751436 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:42.751340    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:43.667877 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:43.667840    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:43.865559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:43.865529    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:43.965407 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:43.965327    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:44.068365 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.068335    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:44.273636 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.273607    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log" limit=1024
Jul 22 01:45:44.346786 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.346752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jwnlg" event={"ID":"52e130bf-16ac-457b-bff4-e49e11b0e3bb","Type":"ContainerStarted","Data":"6c4a96ad669d6a6d1797118c865f1415a239ad524fb2c9e889975d7ad7cebc68"}
Jul 22 01:45:44.346949 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.346805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jwnlg" event={"ID":"52e130bf-16ac-457b-bff4-e49e11b0e3bb","Type":"ContainerStarted","Data":"0cb291eb0766a8ccff636c119fb9c70c681816faf95217bdcc3e985b1044d351"}
Jul 22 01:45:44.346949 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.346874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:44.364851 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.364810    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jwnlg" podStartSLOduration=17.958680175 podStartE2EDuration="20.364796349s" totalImagesPullingTime="2.406116174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:44.363380034 +0000 UTC m=+55.866238264" watchObservedRunningTime="2026-07-22 01:45:44.364796349 +0000 UTC m=+55.867654578"
Jul 22 01:45:44.962393 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.962359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:44.964633 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.964614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32a06c56-9fab-49ba-88ce-6606585a4e0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d7zvh\" (UID: \"32a06c56-9fab-49ba-88ce-6606585a4e0f\") " pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:44.969607 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:44.969589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d7zvh"
Jul 22 01:45:45.089547 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:45.089525    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d7zvh"]
Jul 22 01:45:45.091941 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:45.091913    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a06c56_9fab_49ba_88ce_6606585a4e0f.slice/crio-8abe738806696286c05ff63e072ac34a39af98186ce58be8f10d810382ea5f2a WatchSource:0}: Error finding container 8abe738806696286c05ff63e072ac34a39af98186ce58be8f10d810382ea5f2a: Status 404 returned error can't find the container with id 8abe738806696286c05ff63e072ac34a39af98186ce58be8f10d810382ea5f2a
Jul 22 01:45:45.350660 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:45.350625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7zvh" event={"ID":"32a06c56-9fab-49ba-88ce-6606585a4e0f","Type":"ContainerStarted","Data":"8abe738806696286c05ff63e072ac34a39af98186ce58be8f10d810382ea5f2a"}
Jul 22 01:45:47.102844 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:47.102800    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" podUID="5f8342a5-09b9-46e5-87e0-2b042a9bb940" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 22 01:45:49.261120 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:49.261093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf44z"
Jul 22 01:45:51.374542 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:51.374501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7zvh" event={"ID":"32a06c56-9fab-49ba-88ce-6606585a4e0f","Type":"ContainerStarted","Data":"9430ab24522dc39db7e837a96ccfd30250c917ba40a24c6f778b5d8d71f91ad3"}
Jul 22 01:45:54.256304 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.256273    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f95749bb-srd25"]
Jul 22 01:45:54.314349 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.314316    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66595658cd-zvfqd"]
Jul 22 01:45:54.317121 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.317102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.335743 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.335641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66595658cd-zvfqd"]
Jul 22 01:45:54.353057 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.353034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jwnlg"
Jul 22 01:45:54.386538 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.386505    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7zvh" event={"ID":"32a06c56-9fab-49ba-88ce-6606585a4e0f","Type":"ContainerStarted","Data":"2205e0b3c81cbfeaf298d7d993b550cdca0b3a01da9fdac0e70102fadd197854"}
Jul 22 01:45:54.435780 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.435716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3950bd94-18cd-4904-a452-2b3d4e307da3-trusted-ca\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.435940 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.435798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-bound-sa-token\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.435940 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.435870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3950bd94-18cd-4904-a452-2b3d4e307da3-image-registry-private-configuration\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.436181 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.436155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3950bd94-18cd-4904-a452-2b3d4e307da3-installation-pull-secrets\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.436311 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.436278    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpbc\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-kube-api-access-rlpbc\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.436433 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.436356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3950bd94-18cd-4904-a452-2b3d4e307da3-registry-certificates\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.436433 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.436392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-registry-tls\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.436433 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.436415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3950bd94-18cd-4904-a452-2b3d4e307da3-ca-trust-extracted\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.536934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-registry-tls\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.536982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3950bd94-18cd-4904-a452-2b3d4e307da3-ca-trust-extracted\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3950bd94-18cd-4904-a452-2b3d4e307da3-trusted-ca\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-bound-sa-token\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537249 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3950bd94-18cd-4904-a452-2b3d4e307da3-image-registry-private-configuration\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537774 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537427    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3950bd94-18cd-4904-a452-2b3d4e307da3-installation-pull-secrets\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537774 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3950bd94-18cd-4904-a452-2b3d4e307da3-ca-trust-extracted\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537774 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpbc\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-kube-api-access-rlpbc\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.537774 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.537567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3950bd94-18cd-4904-a452-2b3d4e307da3-registry-certificates\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.538780 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.538754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3950bd94-18cd-4904-a452-2b3d4e307da3-trusted-ca\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.539665 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.539257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3950bd94-18cd-4904-a452-2b3d4e307da3-registry-certificates\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.540805 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.540759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3950bd94-18cd-4904-a452-2b3d4e307da3-image-registry-private-configuration\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.541272 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.541245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-registry-tls\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.541523 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.541500    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3950bd94-18cd-4904-a452-2b3d4e307da3-installation-pull-secrets\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.550987 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.550961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-bound-sa-token\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.551292 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.551272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpbc\" (UniqueName: \"kubernetes.io/projected/3950bd94-18cd-4904-a452-2b3d4e307da3-kube-api-access-rlpbc\") pod \"image-registry-66595658cd-zvfqd\" (UID: \"3950bd94-18cd-4904-a452-2b3d4e307da3\") " pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.628272 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.628240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:54.740015 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.739983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:54.740165 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.740029    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:54.742585 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.742561    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 01:45:54.742704 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.742563    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 01:45:54.753311 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.753257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e297bc0c-22b3-4e81-a4b4-ae416d6ae24e-original-pull-secret\") pod \"global-pull-secret-syncer-sqzb4\" (UID: \"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e\") " pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:54.753641 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.753622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e53db84-b279-44fb-a467-07f335547a01-metrics-certs\") pod \"network-metrics-daemon-rh2rd\" (UID: \"6e53db84-b279-44fb-a467-07f335547a01\") " pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:54.778481 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.778449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66595658cd-zvfqd"]
Jul 22 01:45:54.782132 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:54.782104    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3950bd94_18cd_4904_a452_2b3d4e307da3.slice/crio-918067ebc9955280bad8a9b298728761326b76fbf7c95bee88bcadce8feb75f2 WatchSource:0}: Error finding container 918067ebc9955280bad8a9b298728761326b76fbf7c95bee88bcadce8feb75f2: Status 404 returned error can't find the container with id 918067ebc9955280bad8a9b298728761326b76fbf7c95bee88bcadce8feb75f2
Jul 22 01:45:54.918323 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.918290    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cgfms\""
Jul 22 01:45:54.921427 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.921403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sqzb4"
Jul 22 01:45:54.926491 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.926467    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh2rd"
Jul 22 01:45:54.941512 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.941479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:54.944087 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.944064    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 01:45:54.954665 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.954642    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 01:45:54.966330 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:54.966231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25xws\" (UniqueName: \"kubernetes.io/projected/be905c4d-e0e9-4a07-83fc-7c9063b709d6-kube-api-access-25xws\") pod \"network-check-target-wpwxr\" (UID: \"be905c4d-e0e9-4a07-83fc-7c9063b709d6\") " pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:55.084065 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.084032    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sqzb4"]
Jul 22 01:45:55.088008 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:55.087976    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode297bc0c_22b3_4e81_a4b4_ae416d6ae24e.slice/crio-385934a0fd06d0b2f3239fe1d7950a89d19bf7bc09ecb704e67be7f5ac1dac91 WatchSource:0}: Error finding container 385934a0fd06d0b2f3239fe1d7950a89d19bf7bc09ecb704e67be7f5ac1dac91: Status 404 returned error can't find the container with id 385934a0fd06d0b2f3239fe1d7950a89d19bf7bc09ecb704e67be7f5ac1dac91
Jul 22 01:45:55.107629 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.107596    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh2rd"]
Jul 22 01:45:55.110938 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:55.110910    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e53db84_b279_44fb_a467_07f335547a01.slice/crio-37f3f66febbbf298d17d5bcf2b303809e0c25115591aee7265f87c410d8e6e4e WatchSource:0}: Error finding container 37f3f66febbbf298d17d5bcf2b303809e0c25115591aee7265f87c410d8e6e4e: Status 404 returned error can't find the container with id 37f3f66febbbf298d17d5bcf2b303809e0c25115591aee7265f87c410d8e6e4e
Jul 22 01:45:55.211042 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.211009    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9wgwz\""
Jul 22 01:45:55.219171 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.219131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:45:55.362418 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.362385    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wpwxr"]
Jul 22 01:45:55.368927 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:45:55.368894    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe905c4d_e0e9_4a07_83fc_7c9063b709d6.slice/crio-40aab81c1a87393bc9ef8c76fe722fe19d10baa06cef0e56ce78c3204c017a5e WatchSource:0}: Error finding container 40aab81c1a87393bc9ef8c76fe722fe19d10baa06cef0e56ce78c3204c017a5e: Status 404 returned error can't find the container with id 40aab81c1a87393bc9ef8c76fe722fe19d10baa06cef0e56ce78c3204c017a5e
Jul 22 01:45:55.391790 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.391756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sqzb4" event={"ID":"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e","Type":"ContainerStarted","Data":"385934a0fd06d0b2f3239fe1d7950a89d19bf7bc09ecb704e67be7f5ac1dac91"}
Jul 22 01:45:55.393720 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.393688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66595658cd-zvfqd" event={"ID":"3950bd94-18cd-4904-a452-2b3d4e307da3","Type":"ContainerStarted","Data":"d9192885f03914a313fcc75ee503bebea376cba085365a0b7975c624aed832eb"}
Jul 22 01:45:55.394147 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.394120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:45:55.394236 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.394151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66595658cd-zvfqd" event={"ID":"3950bd94-18cd-4904-a452-2b3d4e307da3","Type":"ContainerStarted","Data":"918067ebc9955280bad8a9b298728761326b76fbf7c95bee88bcadce8feb75f2"}
Jul 22 01:45:55.395218 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.395193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wpwxr" event={"ID":"be905c4d-e0e9-4a07-83fc-7c9063b709d6","Type":"ContainerStarted","Data":"40aab81c1a87393bc9ef8c76fe722fe19d10baa06cef0e56ce78c3204c017a5e"}
Jul 22 01:45:55.397899 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.397874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh2rd" event={"ID":"6e53db84-b279-44fb-a467-07f335547a01","Type":"ContainerStarted","Data":"37f3f66febbbf298d17d5bcf2b303809e0c25115591aee7265f87c410d8e6e4e"}
Jul 22 01:45:55.414488 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:55.414437    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66595658cd-zvfqd" podStartSLOduration=1.414417317 podStartE2EDuration="1.414417317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:45:55.413381241 +0000 UTC m=+66.916239496" watchObservedRunningTime="2026-07-22 01:45:55.414417317 +0000 UTC m=+66.917275548"
Jul 22 01:45:57.102800 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:57.102752    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" podUID="5f8342a5-09b9-46e5-87e0-2b042a9bb940" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 22 01:45:57.103266 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:57.103024    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh"
Jul 22 01:45:57.103719 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:57.103671    2561 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"e76a706b4d254beb158f3aaacfbb948264af7b24a845057951aff145e03581b9"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 22 01:45:57.103829 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:57.103763    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" podUID="5f8342a5-09b9-46e5-87e0-2b042a9bb940" containerName="service-proxy" containerID="cri-o://e76a706b4d254beb158f3aaacfbb948264af7b24a845057951aff145e03581b9" gracePeriod=30
Jul 22 01:45:58.412549 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:58.412509    2561 generic.go:356] "Generic (PLEG): container finished" podID="5f8342a5-09b9-46e5-87e0-2b042a9bb940" containerID="e76a706b4d254beb158f3aaacfbb948264af7b24a845057951aff145e03581b9" exitCode=2
Jul 22 01:45:58.412944 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:45:58.412576    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" event={"ID":"5f8342a5-09b9-46e5-87e0-2b042a9bb940","Type":"ContainerDied","Data":"e76a706b4d254beb158f3aaacfbb948264af7b24a845057951aff145e03581b9"}
Jul 22 01:46:01.425703 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.425659    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh2rd" event={"ID":"6e53db84-b279-44fb-a467-07f335547a01","Type":"ContainerStarted","Data":"fe0a05f14274652186cdef777c5698636775ac4b0428887775c252a93f4279b3"}
Jul 22 01:46:01.426209 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.425778    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh2rd" event={"ID":"6e53db84-b279-44fb-a467-07f335547a01","Type":"ContainerStarted","Data":"0ef7f1eeeac0ebbd001c45e1c20ef47e29df5f16f5e0a9119150d76b3f33cbd7"}
Jul 22 01:46:01.427235 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.427183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sqzb4" event={"ID":"e297bc0c-22b3-4e81-a4b4-ae416d6ae24e","Type":"ContainerStarted","Data":"6683ba372cf8c90cdd83fa51a682eebac03f0ce28222ad555097a7425837ee6b"}
Jul 22 01:46:01.433348 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.433321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b86c9c78c-gsbmh" event={"ID":"5f8342a5-09b9-46e5-87e0-2b042a9bb940","Type":"ContainerStarted","Data":"5b7540d1c22057af592ba630c9cf2e9585549d83111a5c8f74782eed6c75fd01"}
Jul 22 01:46:01.435296 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.435270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d7zvh" event={"ID":"32a06c56-9fab-49ba-88ce-6606585a4e0f","Type":"ContainerStarted","Data":"481b61e52f349c75daa808f173f323aeffcd18a81cba2fc8d317b1cb627034ee"}
Jul 22 01:46:01.436641 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.436624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wpwxr" event={"ID":"be905c4d-e0e9-4a07-83fc-7c9063b709d6","Type":"ContainerStarted","Data":"f83ee2b5bbe6b26ad383115fdd1965c808d7b6fa2cbcf6baa7e6ab195067e18c"}
Jul 22 01:46:01.436768 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.436756    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:46:01.444339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.444292    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rh2rd" podStartSLOduration=66.82655697 podStartE2EDuration="1m12.444275272s" totalImagesPullingTime="5.617718302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:01.443014934 +0000 UTC m=+72.945873172" watchObservedRunningTime="2026-07-22 01:46:01.444275272 +0000 UTC m=+72.947133503"
Jul 22 01:46:01.476903 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.476843    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sqzb4" podStartSLOduration=66.829535672 podStartE2EDuration="1m12.476826249s" totalImagesPullingTime="5.647290577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:01.476501725 +0000 UTC m=+72.979359955" watchObservedRunningTime="2026-07-22 01:46:01.476826249 +0000 UTC m=+72.979684482"
Jul 22 01:46:01.493285 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.493242    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wpwxr" podStartSLOduration=67.127000932 podStartE2EDuration="1m12.493228624s" totalImagesPullingTime="5.366227692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:01.492406643 +0000 UTC m=+72.995264882" watchObservedRunningTime="2026-07-22 01:46:01.493228624 +0000 UTC m=+72.996086887"
Jul 22 01:46:01.512601 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:01.512553    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d7zvh" podStartSLOduration=17.028499562 podStartE2EDuration="32.512537085s" totalImagesPullingTime="15.484037523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 01:45:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:01.511169977 +0000 UTC m=+73.014028205" watchObservedRunningTime="2026-07-22 01:46:01.512537085 +0000 UTC m=+73.015395314"
Jul 22 01:46:04.262420 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:04.262391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:46:15.349015 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.348901    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-97gct"]
Jul 22 01:46:15.355465 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.355443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.358192 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.358172    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 22 01:46:15.358321 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.358278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p24dd\""
Jul 22 01:46:15.358423 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.358302    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 22 01:46:15.358989 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.358971    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 22 01:46:15.359079 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.359020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 22 01:46:15.359282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.359266    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 22 01:46:15.359420 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.359398    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 22 01:46:15.497631 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-root\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.497831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-sys\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.497831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.497831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2kf\" (UniqueName: \"kubernetes.io/projected/d7256c0d-e840-45aa-9964-46cd09750f78-kube-api-access-qp2kf\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.497831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-wtmp\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.497831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-textfile\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.497831 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.498019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-tls\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.498019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.497875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7256c0d-e840-45aa-9964-46cd09750f78-metrics-client-ca\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598219 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2kf\" (UniqueName: \"kubernetes.io/projected/d7256c0d-e840-45aa-9964-46cd09750f78-kube-api-access-qp2kf\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-wtmp\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-textfile\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-tls\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598340    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7256c0d-e840-45aa-9964-46cd09750f78-metrics-client-ca\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598356    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-root\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598397 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-sys\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598793 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-wtmp\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598793 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-sys\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598793 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7256c0d-e840-45aa-9964-46cd09750f78-root\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.598793 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:46:15.598470    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 22 01:46:15.598793 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:46:15.598548    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-tls podName:d7256c0d-e840-45aa-9964-46cd09750f78 nodeName:}" failed. No retries permitted until 2026-07-22 01:46:16.098526485 +0000 UTC m=+87.601384706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-tls") pod "node-exporter-97gct" (UID: "d7256c0d-e840-45aa-9964-46cd09750f78") : secret "node-exporter-tls" not found
Jul 22 01:46:15.598793 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.598694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-textfile\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.599045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.599009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7256c0d-e840-45aa-9964-46cd09750f78-metrics-client-ca\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.599045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.599032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-accelerators-collector-config\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.600483 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.600461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:15.609102 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:15.609082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2kf\" (UniqueName: \"kubernetes.io/projected/d7256c0d-e840-45aa-9964-46cd09750f78-kube-api-access-qp2kf\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:16.101250 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:16.101216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-tls\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:16.103331 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:16.103312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7256c0d-e840-45aa-9964-46cd09750f78-node-exporter-tls\") pod \"node-exporter-97gct\" (UID: \"d7256c0d-e840-45aa-9964-46cd09750f78\") " pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:16.264755 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:16.264717    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-97gct"
Jul 22 01:46:16.278767 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:46:16.278718    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7256c0d_e840_45aa_9964_46cd09750f78.slice/crio-b79f892af2f9d31e25ff21789e904489b4a153e7e4042c67c7d4b5fd552bf1aa WatchSource:0}: Error finding container b79f892af2f9d31e25ff21789e904489b4a153e7e4042c67c7d4b5fd552bf1aa: Status 404 returned error can't find the container with id b79f892af2f9d31e25ff21789e904489b4a153e7e4042c67c7d4b5fd552bf1aa
Jul 22 01:46:16.407005 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:16.406931    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66595658cd-zvfqd"
Jul 22 01:46:16.487012 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:16.486974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97gct" event={"ID":"d7256c0d-e840-45aa-9964-46cd09750f78","Type":"ContainerStarted","Data":"b79f892af2f9d31e25ff21789e904489b4a153e7e4042c67c7d4b5fd552bf1aa"}
Jul 22 01:46:17.492238 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:17.492137    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7256c0d-e840-45aa-9964-46cd09750f78" containerID="7101398afe30262c02530c561b7a0aa5bc07bc0d799b384e92cd62cb645f3dcf" exitCode=0
Jul 22 01:46:17.492238 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:17.492189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97gct" event={"ID":"d7256c0d-e840-45aa-9964-46cd09750f78","Type":"ContainerDied","Data":"7101398afe30262c02530c561b7a0aa5bc07bc0d799b384e92cd62cb645f3dcf"}
Jul 22 01:46:18.498087 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:18.498047    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97gct" event={"ID":"d7256c0d-e840-45aa-9964-46cd09750f78","Type":"ContainerStarted","Data":"9e2bced1612a1c1c3de51d3d67e7a9c0c6eb31504d97f46b5d5f4e50e687aa9b"}
Jul 22 01:46:18.498455 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:18.498110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-97gct" event={"ID":"d7256c0d-e840-45aa-9964-46cd09750f78","Type":"ContainerStarted","Data":"6251ce2f2720303ebbc9f4a4bcbed3b993febba4e37928570f54fba39c7287fe"}
Jul 22 01:46:18.521987 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:18.521942    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-97gct" podStartSLOduration=2.510948403 podStartE2EDuration="3.521927832s" totalImagesPullingTime="949.651403ms" totalInitContainerRuntime="61.328026ms" isStatefulPod=true podCreationTimestamp="2026-07-22 01:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:46:18.520157618 +0000 UTC m=+90.023015846" watchObservedRunningTime="2026-07-22 01:46:18.521927832 +0000 UTC m=+90.024786058"
Jul 22 01:46:19.278108 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.278054    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f95749bb-srd25" podUID="57ffa62a-77ee-4485-b3ac-7e0872408dee" containerName="registry" containerID="cri-o://db35b7cd353e906649339fb97d4607da45aca3092debf9eb40a7f621eaf78cfc" gracePeriod=30
Jul 22 01:46:19.507442 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.507413    2561 generic.go:356] "Generic (PLEG): container finished" podID="57ffa62a-77ee-4485-b3ac-7e0872408dee" containerID="db35b7cd353e906649339fb97d4607da45aca3092debf9eb40a7f621eaf78cfc" exitCode=0
Jul 22 01:46:19.507765 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.507483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f95749bb-srd25" event={"ID":"57ffa62a-77ee-4485-b3ac-7e0872408dee","Type":"ContainerDied","Data":"db35b7cd353e906649339fb97d4607da45aca3092debf9eb40a7f621eaf78cfc"}
Jul 22 01:46:19.507765 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.507553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f95749bb-srd25" event={"ID":"57ffa62a-77ee-4485-b3ac-7e0872408dee","Type":"ContainerDied","Data":"262a0756c409b4db665f9f56ae18e1c12c7ece0fc9dc591ca01bb865bef17778"}
Jul 22 01:46:19.507765 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.507569    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="262a0756c409b4db665f9f56ae18e1c12c7ece0fc9dc591ca01bb865bef17778"
Jul 22 01:46:19.514832 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.514810    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:46:19.627329 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627329 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-trusted-ca\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627329 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-bound-sa-token\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfb5\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-kube-api-access-6lfb5\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627389    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-image-registry-private-configuration\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-installation-pull-secrets\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ffa62a-77ee-4485-b3ac-7e0872408dee-ca-trust-extracted\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627513    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-certificates\") pod \"57ffa62a-77ee-4485-b3ac-7e0872408dee\" (UID: \"57ffa62a-77ee-4485-b3ac-7e0872408dee\") "
Jul 22 01:46:19.627866 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.627724    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:46:19.628412 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.628345    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:46:19.630335 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.630305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:46:19.630437 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.630314    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:46:19.630437 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.630379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-kube-api-access-6lfb5" (OuterVolumeSpecName: "kube-api-access-6lfb5") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "kube-api-access-6lfb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:46:19.630437 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.630420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:46:19.630583 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.630562    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:46:19.636056 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.636029    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ffa62a-77ee-4485-b3ac-7e0872408dee-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "57ffa62a-77ee-4485-b3ac-7e0872408dee" (UID: "57ffa62a-77ee-4485-b3ac-7e0872408dee"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:46:19.728620 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728590    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfb5\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-kube-api-access-6lfb5\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728620 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728616    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-image-registry-private-configuration\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728620 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728625    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57ffa62a-77ee-4485-b3ac-7e0872408dee-installation-pull-secrets\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728841 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728635    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57ffa62a-77ee-4485-b3ac-7e0872408dee-ca-trust-extracted\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728841 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728644    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-certificates\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728841 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728653    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-registry-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728841 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728661    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57ffa62a-77ee-4485-b3ac-7e0872408dee-trusted-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:19.728841 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:19.728669    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57ffa62a-77ee-4485-b3ac-7e0872408dee-bound-sa-token\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:46:20.511643 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:20.511613    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f95749bb-srd25"
Jul 22 01:46:20.553051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:20.553024    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f95749bb-srd25"]
Jul 22 01:46:20.557300 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:20.557268    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f95749bb-srd25"]
Jul 22 01:46:21.101244 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:21.101212    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ffa62a-77ee-4485-b3ac-7e0872408dee" path="/var/lib/kubelet/pods/57ffa62a-77ee-4485-b3ac-7e0872408dee/volumes"
Jul 22 01:46:32.443425 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:46:32.443394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wpwxr"
Jul 22 01:49:25.163681 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.163592    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"]
Jul 22 01:49:25.164152 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.163885    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ffa62a-77ee-4485-b3ac-7e0872408dee" containerName="registry"
Jul 22 01:49:25.164152 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.163901    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ffa62a-77ee-4485-b3ac-7e0872408dee" containerName="registry"
Jul 22 01:49:25.164152 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.163948    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ffa62a-77ee-4485-b3ac-7e0872408dee" containerName="registry"
Jul 22 01:49:25.166608 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.166592    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.169391 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.169373    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 22 01:49:25.169854 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.169840    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 22 01:49:25.169919 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.169894    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 22 01:49:25.170156 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.170143    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 22 01:49:25.170507 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.170493    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ghhvt\""
Jul 22 01:49:25.185694 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.185674    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"]
Jul 22 01:49:25.195881 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.195861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7138d62-ba39-428d-b2d6-845706592431-webhook-cert\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.195973 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.195888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7138d62-ba39-428d-b2d6-845706592431-manifests\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.195973 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.195917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxjnd\" (UniqueName: \"kubernetes.io/projected/d7138d62-ba39-428d-b2d6-845706592431-kube-api-access-wxjnd\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.195973 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.195940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7138d62-ba39-428d-b2d6-845706592431-tmp\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.196074 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.195980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7138d62-ba39-428d-b2d6-845706592431-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.296782 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.296747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7138d62-ba39-428d-b2d6-845706592431-webhook-cert\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.296934 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.296788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7138d62-ba39-428d-b2d6-845706592431-manifests\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.296934 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.296829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxjnd\" (UniqueName: \"kubernetes.io/projected/d7138d62-ba39-428d-b2d6-845706592431-kube-api-access-wxjnd\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.296934 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.296855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7138d62-ba39-428d-b2d6-845706592431-tmp\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.297094 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.296991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7138d62-ba39-428d-b2d6-845706592431-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.297309 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.297282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7138d62-ba39-428d-b2d6-845706592431-manifests\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.299142 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.299117    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7138d62-ba39-428d-b2d6-845706592431-tmp\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.299234 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.299180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7138d62-ba39-428d-b2d6-845706592431-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.299274 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.299245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7138d62-ba39-428d-b2d6-845706592431-webhook-cert\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.306663 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.306643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxjnd\" (UniqueName: \"kubernetes.io/projected/d7138d62-ba39-428d-b2d6-845706592431-kube-api-access-wxjnd\") pod \"opendatahub-operator-controller-manager-585784b644-gdm4q\" (UID: \"d7138d62-ba39-428d-b2d6-845706592431\") " pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.476703 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.476622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:25.605383 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:25.605360    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"]
Jul 22 01:49:25.607542 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:49:25.607520    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7138d62_ba39_428d_b2d6_845706592431.slice/crio-81c34b725b8f69baebd5a486711a295c9483baef299b8ddb4b6ca3807af8b614 WatchSource:0}: Error finding container 81c34b725b8f69baebd5a486711a295c9483baef299b8ddb4b6ca3807af8b614: Status 404 returned error can't find the container with id 81c34b725b8f69baebd5a486711a295c9483baef299b8ddb4b6ca3807af8b614
Jul 22 01:49:26.133338 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:26.133299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q" event={"ID":"d7138d62-ba39-428d-b2d6-845706592431","Type":"ContainerStarted","Data":"81c34b725b8f69baebd5a486711a295c9483baef299b8ddb4b6ca3807af8b614"}
Jul 22 01:49:29.145858 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:29.145826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q" event={"ID":"d7138d62-ba39-428d-b2d6-845706592431","Type":"ContainerStarted","Data":"602d63b1e49e50dd8ce0f46a10a016a73d86abd84064f2a9a513f3c6f097d81e"}
Jul 22 01:49:30.150795 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:30.150759    2561 generic.go:356] "Generic (PLEG): container finished" podID="d7138d62-ba39-428d-b2d6-845706592431" containerID="602d63b1e49e50dd8ce0f46a10a016a73d86abd84064f2a9a513f3c6f097d81e" exitCode=0
Jul 22 01:49:30.151170 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:30.150841    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q" event={"ID":"d7138d62-ba39-428d-b2d6-845706592431","Type":"ContainerDied","Data":"602d63b1e49e50dd8ce0f46a10a016a73d86abd84064f2a9a513f3c6f097d81e"}
Jul 22 01:49:31.155398 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:31.155360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q" event={"ID":"d7138d62-ba39-428d-b2d6-845706592431","Type":"ContainerStarted","Data":"2227722ae0a2b27f919fde04710e06b90506ade555cc3cb99d8bbb23aebe39eb"}
Jul 22 01:49:31.155876 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:31.155480    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:31.178861 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:31.178809    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q" podStartSLOduration=1.8585111539999999 podStartE2EDuration="6.178792985s" totalImagesPullingTime="3.277077551s" totalInitContainerRuntime="1.04320428s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:31.176435435 +0000 UTC m=+282.679293665" watchObservedRunningTime="2026-07-22 01:49:31.178792985 +0000 UTC m=+282.681651215"
Jul 22 01:49:42.161331 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:42.161301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-585784b644-gdm4q"
Jul 22 01:49:44.786096 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.786061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"]
Jul 22 01:49:44.791861 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.791841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.796313 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.796292    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 22 01:49:44.796536 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.796522    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 22 01:49:44.797351 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.797330    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 22 01:49:44.797442 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.797354    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5mjjm\""
Jul 22 01:49:44.797442 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.797363    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 22 01:49:44.811343 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.811322    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"]
Jul 22 01:49:44.825566 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.825536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-557pd\" (UniqueName: \"kubernetes.io/projected/f9d366fd-d3b6-4c08-a513-296e9b2f991e-kube-api-access-557pd\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.825682 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.825574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d366fd-d3b6-4c08-a513-296e9b2f991e-tls-certs\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.825682 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.825603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d366fd-d3b6-4c08-a513-296e9b2f991e-tmp\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.926360 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.926333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-557pd\" (UniqueName: \"kubernetes.io/projected/f9d366fd-d3b6-4c08-a513-296e9b2f991e-kube-api-access-557pd\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.926533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.926370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d366fd-d3b6-4c08-a513-296e9b2f991e-tls-certs\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.926533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.926396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d366fd-d3b6-4c08-a513-296e9b2f991e-tmp\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.928513 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.928493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9d366fd-d3b6-4c08-a513-296e9b2f991e-tmp\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.928771 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.928752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d366fd-d3b6-4c08-a513-296e9b2f991e-tls-certs\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:44.943566 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:44.943537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-557pd\" (UniqueName: \"kubernetes.io/projected/f9d366fd-d3b6-4c08-a513-296e9b2f991e-kube-api-access-557pd\") pod \"kube-auth-proxy-55d847747c-xdp4r\" (UID: \"f9d366fd-d3b6-4c08-a513-296e9b2f991e\") " pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:45.100493 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.100414    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"
Jul 22 01:49:45.230206 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.230158    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55d847747c-xdp4r"]
Jul 22 01:49:45.232440 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:49:45.232410    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d366fd_d3b6_4c08_a513_296e9b2f991e.slice/crio-101ffc9d81bbb1e75a1fafe2451d6a6a4697bde2c3c7fc1b00c462ee9e85d1be WatchSource:0}: Error finding container 101ffc9d81bbb1e75a1fafe2451d6a6a4697bde2c3c7fc1b00c462ee9e85d1be: Status 404 returned error can't find the container with id 101ffc9d81bbb1e75a1fafe2451d6a6a4697bde2c3c7fc1b00c462ee9e85d1be
Jul 22 01:49:45.772508 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.772478    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ns8d9"]
Jul 22 01:49:45.776350 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.776334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:45.778742 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.778712    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 22 01:49:45.787044 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.787021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ns8d9"]
Jul 22 01:49:45.832975 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.832945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb02595-7f25-4b63-aabc-d0a01f48cd40-tls-certs\") pod \"model-serving-api-54d89bd5fc-ns8d9\" (UID: \"4fb02595-7f25-4b63-aabc-d0a01f48cd40\") " pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:45.833116 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.832980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmn8\" (UniqueName: \"kubernetes.io/projected/4fb02595-7f25-4b63-aabc-d0a01f48cd40-kube-api-access-4nmn8\") pod \"model-serving-api-54d89bd5fc-ns8d9\" (UID: \"4fb02595-7f25-4b63-aabc-d0a01f48cd40\") " pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:45.934420 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.934291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb02595-7f25-4b63-aabc-d0a01f48cd40-tls-certs\") pod \"model-serving-api-54d89bd5fc-ns8d9\" (UID: \"4fb02595-7f25-4b63-aabc-d0a01f48cd40\") " pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:45.934420 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.934343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmn8\" (UniqueName: \"kubernetes.io/projected/4fb02595-7f25-4b63-aabc-d0a01f48cd40-kube-api-access-4nmn8\") pod \"model-serving-api-54d89bd5fc-ns8d9\" (UID: \"4fb02595-7f25-4b63-aabc-d0a01f48cd40\") " pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:45.937111 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.937088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb02595-7f25-4b63-aabc-d0a01f48cd40-tls-certs\") pod \"model-serving-api-54d89bd5fc-ns8d9\" (UID: \"4fb02595-7f25-4b63-aabc-d0a01f48cd40\") " pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:45.949943 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:45.949915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmn8\" (UniqueName: \"kubernetes.io/projected/4fb02595-7f25-4b63-aabc-d0a01f48cd40-kube-api-access-4nmn8\") pod \"model-serving-api-54d89bd5fc-ns8d9\" (UID: \"4fb02595-7f25-4b63-aabc-d0a01f48cd40\") " pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:46.023215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.023131    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"]
Jul 22 01:49:46.026673 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.026647    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.030563 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.030433    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 22 01:49:46.030563 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.030458    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 22 01:49:46.030563 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.030518    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 22 01:49:46.030818 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.030628    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2xvbm\""
Jul 22 01:49:46.030818 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.030778    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 22 01:49:46.040783 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.040713    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"]
Jul 22 01:49:46.086184 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.086151    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:46.135599 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-token\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.135783 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.135783 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.135783 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6k7n\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-kube-api-access-f6k7n\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.135783 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-cacerts\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.135783 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.136046 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.135793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-local-certs\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.209484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.209455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r" event={"ID":"f9d366fd-d3b6-4c08-a513-296e9b2f991e","Type":"ContainerStarted","Data":"101ffc9d81bbb1e75a1fafe2451d6a6a4697bde2c3c7fc1b00c462ee9e85d1be"}
Jul 22 01:49:46.215140 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.214983    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ns8d9"]
Jul 22 01:49:46.217256 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:49:46.217234    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb02595_7f25_4b63_aabc_d0a01f48cd40.slice/crio-f5d9ae6cc568a83bd52db63a150c733fb86aa929e9a22042682dc2572524fda5 WatchSource:0}: Error finding container f5d9ae6cc568a83bd52db63a150c733fb86aa929e9a22042682dc2572524fda5: Status 404 returned error can't find the container with id f5d9ae6cc568a83bd52db63a150c733fb86aa929e9a22042682dc2572524fda5
Jul 22 01:49:46.237285 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-token\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.237396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.237396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.237396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6k7n\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-kube-api-access-f6k7n\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.237396 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-cacerts\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.237590 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.237590 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.237443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-local-certs\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.238242 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.238197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.239998 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.239975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-local-certs\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.240266 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.240220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.240372 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.240353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-cacerts\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.240503 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.240483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.245215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.245198    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-token\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.245545 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.245512    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6k7n\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-kube-api-access-f6k7n\") pod \"istiod-openshift-gateway-7574459656-6n5qp\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.338268 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.338188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:46.474691 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:46.474665    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"]
Jul 22 01:49:46.476852 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:49:46.476823    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdf4d84_98d0_4279_93b7_2cac190bb4b0.slice/crio-096953e0b25c17c3f96ea646559c8ba783e26ee190b007c4c1dcf27f435a1787 WatchSource:0}: Error finding container 096953e0b25c17c3f96ea646559c8ba783e26ee190b007c4c1dcf27f435a1787: Status 404 returned error can't find the container with id 096953e0b25c17c3f96ea646559c8ba783e26ee190b007c4c1dcf27f435a1787
Jul 22 01:49:47.215797 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:47.215686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" event={"ID":"dbdf4d84-98d0-4279-93b7-2cac190bb4b0","Type":"ContainerStarted","Data":"096953e0b25c17c3f96ea646559c8ba783e26ee190b007c4c1dcf27f435a1787"}
Jul 22 01:49:47.217147 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:47.217116    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9" event={"ID":"4fb02595-7f25-4b63-aabc-d0a01f48cd40","Type":"ContainerStarted","Data":"f5d9ae6cc568a83bd52db63a150c733fb86aa929e9a22042682dc2572524fda5"}
Jul 22 01:49:48.933036 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:48.933007    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:49:48.940911 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:48.940889    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 22 01:49:48.943401 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:48.943381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:49:49.227632 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:49.227451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9" event={"ID":"4fb02595-7f25-4b63-aabc-d0a01f48cd40","Type":"ContainerStarted","Data":"d867fab326689bc7bde6980e66ef65b862e03a521e5f9296c38955be57ad3e28"}
Jul 22 01:49:49.227632 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:49.227528    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:49:49.253689 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:49.229152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r" event={"ID":"f9d366fd-d3b6-4c08-a513-296e9b2f991e","Type":"ContainerStarted","Data":"bc551f2488e7986b10624d55b403eb9514496d3b77044d023cc6396eb8ad30cf"}
Jul 22 01:49:49.253689 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:49.245061    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9" podStartSLOduration=2.298108699 podStartE2EDuration="4.245047777s" totalImagesPullingTime="1.946939078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:49.243493802 +0000 UTC m=+300.746352046" watchObservedRunningTime="2026-07-22 01:49:49.245047777 +0000 UTC m=+300.747906006"
Jul 22 01:49:49.260570 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:49.260529    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55d847747c-xdp4r" podStartSLOduration=2.328653396 podStartE2EDuration="5.260514539s" totalImagesPullingTime="2.931861143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:49.259231517 +0000 UTC m=+300.762089763" watchObservedRunningTime="2026-07-22 01:49:49.260514539 +0000 UTC m=+300.763372770"
Jul 22 01:49:50.552915 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:50.552872    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 22 01:49:50.553189 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:50.552948    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 22 01:49:51.237712 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:51.237671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" event={"ID":"dbdf4d84-98d0-4279-93b7-2cac190bb4b0","Type":"ContainerStarted","Data":"0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d"}
Jul 22 01:49:51.237910 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:51.237889    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:51.239400 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:51.239371    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-6n5qp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 01:49:51.239520 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:51.239416    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" podUID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 01:49:51.264280 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:51.264235    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" podStartSLOduration=1.190366866 podStartE2EDuration="5.264221104s" totalImagesPullingTime="4.073854238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:51.262261902 +0000 UTC m=+302.765120131" watchObservedRunningTime="2026-07-22 01:49:51.264221104 +0000 UTC m=+302.767079333"
Jul 22 01:49:52.241707 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:52.241676    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:55.387545 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.387514    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"]
Jul 22 01:49:55.390687 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.390665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:55.393407 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.393388    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 22 01:49:55.393717 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.393698    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6p9gl\""
Jul 22 01:49:55.402952 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.402933    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"]
Jul 22 01:49:55.405834 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.405816    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"]
Jul 22 01:49:55.405912 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.405891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.408290 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.408272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 22 01:49:55.408394 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.408276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xknqb\""
Jul 22 01:49:55.417774 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.417756    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"]
Jul 22 01:49:55.517427 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.517387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6vb\" (UniqueName: \"kubernetes.io/projected/8f01f09c-61ed-4df8-aa8c-499ec8785950-kube-api-access-pz6vb\") pod \"llmisvc-controller-manager-777c6bd7d5-4dk5f\" (UID: \"8f01f09c-61ed-4df8-aa8c-499ec8785950\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.517605 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.517447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2984e018-c6fe-45ee-b3c9-e32cd38b2841-cert\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:55.517605 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.517474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f01f09c-61ed-4df8-aa8c-499ec8785950-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4dk5f\" (UID: \"8f01f09c-61ed-4df8-aa8c-499ec8785950\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.517605 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.517511    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62qfr\" (UniqueName: \"kubernetes.io/projected/2984e018-c6fe-45ee-b3c9-e32cd38b2841-kube-api-access-62qfr\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:55.618720 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.618685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2984e018-c6fe-45ee-b3c9-e32cd38b2841-cert\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:55.618720 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.618723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f01f09c-61ed-4df8-aa8c-499ec8785950-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4dk5f\" (UID: \"8f01f09c-61ed-4df8-aa8c-499ec8785950\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.618985 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.618794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62qfr\" (UniqueName: \"kubernetes.io/projected/2984e018-c6fe-45ee-b3c9-e32cd38b2841-kube-api-access-62qfr\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:55.618985 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.618829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6vb\" (UniqueName: \"kubernetes.io/projected/8f01f09c-61ed-4df8-aa8c-499ec8785950-kube-api-access-pz6vb\") pod \"llmisvc-controller-manager-777c6bd7d5-4dk5f\" (UID: \"8f01f09c-61ed-4df8-aa8c-499ec8785950\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.618985 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:49:55.618842    2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 22 01:49:55.618985 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:49:55.618901    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2984e018-c6fe-45ee-b3c9-e32cd38b2841-cert podName:2984e018-c6fe-45ee-b3c9-e32cd38b2841 nodeName:}" failed. No retries permitted until 2026-07-22 01:49:56.118884026 +0000 UTC m=+307.621742236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2984e018-c6fe-45ee-b3c9-e32cd38b2841-cert") pod "kserve-controller-manager-d6f4b65b8-nf8ht" (UID: "2984e018-c6fe-45ee-b3c9-e32cd38b2841") : secret "kserve-webhook-server-cert" not found
Jul 22 01:49:55.621253 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.621231    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f01f09c-61ed-4df8-aa8c-499ec8785950-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4dk5f\" (UID: \"8f01f09c-61ed-4df8-aa8c-499ec8785950\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.627886 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.627865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62qfr\" (UniqueName: \"kubernetes.io/projected/2984e018-c6fe-45ee-b3c9-e32cd38b2841-kube-api-access-62qfr\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:55.631027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.631003    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6vb\" (UniqueName: \"kubernetes.io/projected/8f01f09c-61ed-4df8-aa8c-499ec8785950-kube-api-access-pz6vb\") pod \"llmisvc-controller-manager-777c6bd7d5-4dk5f\" (UID: \"8f01f09c-61ed-4df8-aa8c-499ec8785950\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.715196 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.715095    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:49:55.860015 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.859984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"]
Jul 22 01:49:55.862309 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:49:55.862281    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8f01f09c_61ed_4df8_aa8c_499ec8785950.slice/crio-8229eeb813b93e4ece2261a6a3e1ec20962e20d389217deda8e7d79ebf96dacf WatchSource:0}: Error finding container 8229eeb813b93e4ece2261a6a3e1ec20962e20d389217deda8e7d79ebf96dacf: Status 404 returned error can't find the container with id 8229eeb813b93e4ece2261a6a3e1ec20962e20d389217deda8e7d79ebf96dacf
Jul 22 01:49:55.863524 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:55.863508    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 01:49:56.121635 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:56.121604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2984e018-c6fe-45ee-b3c9-e32cd38b2841-cert\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:56.124005 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:56.123979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2984e018-c6fe-45ee-b3c9-e32cd38b2841-cert\") pod \"kserve-controller-manager-d6f4b65b8-nf8ht\" (UID: \"2984e018-c6fe-45ee-b3c9-e32cd38b2841\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:56.255754 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:56.255712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f" event={"ID":"8f01f09c-61ed-4df8-aa8c-499ec8785950","Type":"ContainerStarted","Data":"8229eeb813b93e4ece2261a6a3e1ec20962e20d389217deda8e7d79ebf96dacf"}
Jul 22 01:49:56.300084 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:56.300056    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:56.432163 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:56.432138    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"]
Jul 22 01:49:56.434216 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:49:56.434191    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2984e018_c6fe_45ee_b3c9_e32cd38b2841.slice/crio-a203cc6cd15b0b158513895b5292e07083bbcd7455312df792efcfcc0c4036d7 WatchSource:0}: Error finding container a203cc6cd15b0b158513895b5292e07083bbcd7455312df792efcfcc0c4036d7: Status 404 returned error can't find the container with id a203cc6cd15b0b158513895b5292e07083bbcd7455312df792efcfcc0c4036d7
Jul 22 01:49:57.262412 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:57.262371    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht" event={"ID":"2984e018-c6fe-45ee-b3c9-e32cd38b2841","Type":"ContainerStarted","Data":"a203cc6cd15b0b158513895b5292e07083bbcd7455312df792efcfcc0c4036d7"}
Jul 22 01:49:59.210384 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.210356    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"]
Jul 22 01:49:59.210690 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.210572    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" podUID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" containerName="discovery" containerID="cri-o://0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d" gracePeriod=30
Jul 22 01:49:59.272359 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.272328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht" event={"ID":"2984e018-c6fe-45ee-b3c9-e32cd38b2841","Type":"ContainerStarted","Data":"39a45eee900238b4d9168b4ec7ff5d6b9baa2707a30b0f3d95e2f5b1e72290e8"}
Jul 22 01:49:59.272481 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.272461    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:49:59.291749 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.291682    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht" podStartSLOduration=1.586722693 podStartE2EDuration="4.291666961s" totalImagesPullingTime="2.704944268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:49:59.290651547 +0000 UTC m=+310.793509825" watchObservedRunningTime="2026-07-22 01:49:59.291666961 +0000 UTC m=+310.794525190"
Jul 22 01:49:59.456442 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.456417    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:49:59.550533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550507    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-token\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.550533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550541    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-ca-configmap\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.550794 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-kubeconfig\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.550794 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6k7n\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-kube-api-access-f6k7n\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.550794 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-cacerts\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.550794 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550748    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-local-certs\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.550794 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550778    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-dns-cert\") pod \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\" (UID: \"dbdf4d84-98d0-4279-93b7-2cac190bb4b0\") "
Jul 22 01:49:59.551051 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.550984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:49:59.552995 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.552972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:49:59.553088 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.553005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-local-certs" (OuterVolumeSpecName: "local-certs") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:49:59.553193 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.553171    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-cacerts" (OuterVolumeSpecName: "cacerts") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:49:59.553193 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.553183    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:49:59.553287 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.553195    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:49:59.553287 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.553202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-kube-api-access-f6k7n" (OuterVolumeSpecName: "kube-api-access-f6k7n") pod "dbdf4d84-98d0-4279-93b7-2cac190bb4b0" (UID: "dbdf4d84-98d0-4279-93b7-2cac190bb4b0"). InnerVolumeSpecName "kube-api-access-f6k7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:49:59.651706 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651678    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-token\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:49:59.651706 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651704    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-ca-configmap\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:49:59.651885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651714    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-kubeconfig\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:49:59.651885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651723    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6k7n\" (UniqueName: \"kubernetes.io/projected/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-kube-api-access-f6k7n\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:49:59.651885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651752    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-cacerts\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:49:59.651885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651761    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-local-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:49:59.651885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:49:59.651769    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbdf4d84-98d0-4279-93b7-2cac190bb4b0-istio-csr-dns-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:50:00.237215 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.237185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ns8d9"
Jul 22 01:50:00.278420 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.278380    2561 generic.go:356] "Generic (PLEG): container finished" podID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" containerID="0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d" exitCode=0
Jul 22 01:50:00.278592 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.278424    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"
Jul 22 01:50:00.278592 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.278457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" event={"ID":"dbdf4d84-98d0-4279-93b7-2cac190bb4b0","Type":"ContainerDied","Data":"0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d"}
Jul 22 01:50:00.278592 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.278508    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp" event={"ID":"dbdf4d84-98d0-4279-93b7-2cac190bb4b0","Type":"ContainerDied","Data":"096953e0b25c17c3f96ea646559c8ba783e26ee190b007c4c1dcf27f435a1787"}
Jul 22 01:50:00.278592 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.278536    2561 scope.go:117] "RemoveContainer" containerID="0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d"
Jul 22 01:50:00.280300 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.280276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f" event={"ID":"8f01f09c-61ed-4df8-aa8c-499ec8785950","Type":"ContainerStarted","Data":"6ccb0da3c89ede96953c18d81c6114378d7418ccd5d770c87911bef070d2b8ed"}
Jul 22 01:50:00.280389 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.280343    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:50:00.287897 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.287876    2561 scope.go:117] "RemoveContainer" containerID="0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d"
Jul 22 01:50:00.288165 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:50:00.288145    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d\": container with ID starting with 0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d not found: ID does not exist" containerID="0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d"
Jul 22 01:50:00.288254 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.288169    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d"} err="failed to get container status \"0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d\": rpc error: code = NotFound desc = could not find container \"0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d\": container with ID starting with 0b0f1c4708fa5d49335c7625b39b20084c00a563fc1d5e0b1fe3fa0d487cd16d not found: ID does not exist"
Jul 22 01:50:00.298162 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.298100    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f" podStartSLOduration=2.023005699 podStartE2EDuration="5.298087813s" totalImagesPullingTime="3.275082114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:50:00.297235189 +0000 UTC m=+311.800093421" watchObservedRunningTime="2026-07-22 01:50:00.298087813 +0000 UTC m=+311.800946041"
Jul 22 01:50:00.310620 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.310596    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"]
Jul 22 01:50:00.316596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:00.316573    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6n5qp"]
Jul 22 01:50:01.101164 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:01.101132    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" path="/var/lib/kubelet/pods/dbdf4d84-98d0-4279-93b7-2cac190bb4b0/volumes"
Jul 22 01:50:30.285388 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:30.285361    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nf8ht"
Jul 22 01:50:31.292803 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:31.292769    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4dk5f"
Jul 22 01:50:45.921526 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.921441    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9wmp"]
Jul 22 01:50:45.921984 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.921719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" containerName="discovery"
Jul 22 01:50:45.921984 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.921743    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" containerName="discovery"
Jul 22 01:50:45.921984 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.921787    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdf4d84-98d0-4279-93b7-2cac190bb4b0" containerName="discovery"
Jul 22 01:50:45.924721 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.924703    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:45.927403 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.927376    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-fhhgr\""
Jul 22 01:50:45.928379 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.928355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 22 01:50:45.928481 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.928379    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 22 01:50:45.934521 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:45.934484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9wmp"]
Jul 22 01:50:46.085188 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.085154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqsh\" (UniqueName: \"kubernetes.io/projected/4354f739-ab3a-42e1-8348-dc6df37f0287-kube-api-access-vvqsh\") pod \"kuadrant-operator-catalog-k9wmp\" (UID: \"4354f739-ab3a-42e1-8348-dc6df37f0287\") " pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:46.186282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.186198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqsh\" (UniqueName: \"kubernetes.io/projected/4354f739-ab3a-42e1-8348-dc6df37f0287-kube-api-access-vvqsh\") pod \"kuadrant-operator-catalog-k9wmp\" (UID: \"4354f739-ab3a-42e1-8348-dc6df37f0287\") " pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:46.193988 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.193961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqsh\" (UniqueName: \"kubernetes.io/projected/4354f739-ab3a-42e1-8348-dc6df37f0287-kube-api-access-vvqsh\") pod \"kuadrant-operator-catalog-k9wmp\" (UID: \"4354f739-ab3a-42e1-8348-dc6df37f0287\") " pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:46.235981 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.235958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:46.362559 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.362529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9wmp"]
Jul 22 01:50:46.365378 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:50:46.365351    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4354f739_ab3a_42e1_8348_dc6df37f0287.slice/crio-3057fa13c7c757140e681cc5ff19d1542cb2d1861becf2c6101f0e54d479629a WatchSource:0}: Error finding container 3057fa13c7c757140e681cc5ff19d1542cb2d1861becf2c6101f0e54d479629a: Status 404 returned error can't find the container with id 3057fa13c7c757140e681cc5ff19d1542cb2d1861becf2c6101f0e54d479629a
Jul 22 01:50:46.425012 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.424966    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9wmp"]
Jul 22 01:50:46.450893 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.450826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp" event={"ID":"4354f739-ab3a-42e1-8348-dc6df37f0287","Type":"ContainerStarted","Data":"3057fa13c7c757140e681cc5ff19d1542cb2d1861becf2c6101f0e54d479629a"}
Jul 22 01:50:46.495535 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.495510    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tn7qs"]
Jul 22 01:50:46.499596 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.499581    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:46.507220 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.507201    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tn7qs"]
Jul 22 01:50:46.691309 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.691281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4gf\" (UniqueName: \"kubernetes.io/projected/73da48b7-8657-4665-b5c7-8615a13e790e-kube-api-access-xr4gf\") pod \"kuadrant-operator-catalog-tn7qs\" (UID: \"73da48b7-8657-4665-b5c7-8615a13e790e\") " pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:46.792670 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.792638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4gf\" (UniqueName: \"kubernetes.io/projected/73da48b7-8657-4665-b5c7-8615a13e790e-kube-api-access-xr4gf\") pod \"kuadrant-operator-catalog-tn7qs\" (UID: \"73da48b7-8657-4665-b5c7-8615a13e790e\") " pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:46.800585 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.800562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4gf\" (UniqueName: \"kubernetes.io/projected/73da48b7-8657-4665-b5c7-8615a13e790e-kube-api-access-xr4gf\") pod \"kuadrant-operator-catalog-tn7qs\" (UID: \"73da48b7-8657-4665-b5c7-8615a13e790e\") " pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:46.810048 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.810028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:46.935905 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:46.935882    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tn7qs"]
Jul 22 01:50:46.938249 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:50:46.938206    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73da48b7_8657_4665_b5c7_8615a13e790e.slice/crio-8b6fd966a77b42c72761adab985b9e0be4218368aed391b68867ba85a04f0fd2 WatchSource:0}: Error finding container 8b6fd966a77b42c72761adab985b9e0be4218368aed391b68867ba85a04f0fd2: Status 404 returned error can't find the container with id 8b6fd966a77b42c72761adab985b9e0be4218368aed391b68867ba85a04f0fd2
Jul 22 01:50:47.457989 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:47.457948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs" event={"ID":"73da48b7-8657-4665-b5c7-8615a13e790e","Type":"ContainerStarted","Data":"8b6fd966a77b42c72761adab985b9e0be4218368aed391b68867ba85a04f0fd2"}
Jul 22 01:50:49.467091 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.467056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs" event={"ID":"73da48b7-8657-4665-b5c7-8615a13e790e","Type":"ContainerStarted","Data":"11f152b600ea50028d831f97743ab51559ac0c7e61f8ea156c220f82b5114e07"}
Jul 22 01:50:49.468484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.468463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp" event={"ID":"4354f739-ab3a-42e1-8348-dc6df37f0287","Type":"ContainerStarted","Data":"5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030"}
Jul 22 01:50:49.468592 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.468567    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp" podUID="4354f739-ab3a-42e1-8348-dc6df37f0287" containerName="registry-server" containerID="cri-o://5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030" gracePeriod=2
Jul 22 01:50:49.481619 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.481571    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs" podStartSLOduration=1.997226107 podStartE2EDuration="3.481559282s" totalImagesPullingTime="1.484333175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:50:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:50:49.481370097 +0000 UTC m=+360.984228327" watchObservedRunningTime="2026-07-22 01:50:49.481559282 +0000 UTC m=+360.984417511"
Jul 22 01:50:49.495876 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.495842    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp" podStartSLOduration=2.441588774 podStartE2EDuration="4.495830104s" totalImagesPullingTime="2.05424133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:50:49.495004615 +0000 UTC m=+360.997862845" watchObservedRunningTime="2026-07-22 01:50:49.495830104 +0000 UTC m=+360.998688333"
Jul 22 01:50:49.702842 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.702817    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:49.814694 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.814666    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvqsh\" (UniqueName: \"kubernetes.io/projected/4354f739-ab3a-42e1-8348-dc6df37f0287-kube-api-access-vvqsh\") pod \"4354f739-ab3a-42e1-8348-dc6df37f0287\" (UID: \"4354f739-ab3a-42e1-8348-dc6df37f0287\") "
Jul 22 01:50:49.816704 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.816680    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4354f739-ab3a-42e1-8348-dc6df37f0287-kube-api-access-vvqsh" (OuterVolumeSpecName: "kube-api-access-vvqsh") pod "4354f739-ab3a-42e1-8348-dc6df37f0287" (UID: "4354f739-ab3a-42e1-8348-dc6df37f0287"). InnerVolumeSpecName "kube-api-access-vvqsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:50:49.915621 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:49.915598    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvqsh\" (UniqueName: \"kubernetes.io/projected/4354f739-ab3a-42e1-8348-dc6df37f0287-kube-api-access-vvqsh\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:50:50.473204 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.473167    2561 generic.go:356] "Generic (PLEG): container finished" podID="4354f739-ab3a-42e1-8348-dc6df37f0287" containerID="5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030" exitCode=0
Jul 22 01:50:50.473692 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.473231    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp"
Jul 22 01:50:50.473692 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.473250    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp" event={"ID":"4354f739-ab3a-42e1-8348-dc6df37f0287","Type":"ContainerDied","Data":"5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030"}
Jul 22 01:50:50.473692 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.473298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k9wmp" event={"ID":"4354f739-ab3a-42e1-8348-dc6df37f0287","Type":"ContainerDied","Data":"3057fa13c7c757140e681cc5ff19d1542cb2d1861becf2c6101f0e54d479629a"}
Jul 22 01:50:50.473692 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.473317    2561 scope.go:117] "RemoveContainer" containerID="5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030"
Jul 22 01:50:50.482002 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.481986    2561 scope.go:117] "RemoveContainer" containerID="5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030"
Jul 22 01:50:50.482242 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:50:50.482225    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030\": container with ID starting with 5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030 not found: ID does not exist" containerID="5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030"
Jul 22 01:50:50.482285 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.482251    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030"} err="failed to get container status \"5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030\": rpc error: code = NotFound desc = could not find container \"5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030\": container with ID starting with 5f0efc2142c7acedcafd4b67bb9f6e00fa59e6e404d74a3318c1a3d8053f9030 not found: ID does not exist"
Jul 22 01:50:50.493575 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.493553    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9wmp"]
Jul 22 01:50:50.497447 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:50.497429    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k9wmp"]
Jul 22 01:50:51.100654 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:51.100622    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4354f739-ab3a-42e1-8348-dc6df37f0287" path="/var/lib/kubelet/pods/4354f739-ab3a-42e1-8348-dc6df37f0287/volumes"
Jul 22 01:50:56.810871 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:56.810841    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:56.811279 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:56.810891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:56.834957 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:56.834934    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:50:57.521864 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:50:57.521836    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tn7qs"
Jul 22 01:51:20.036182 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.036151    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"]
Jul 22 01:51:20.036788 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.036537    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4354f739-ab3a-42e1-8348-dc6df37f0287" containerName="registry-server"
Jul 22 01:51:20.036788 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.036553    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4354f739-ab3a-42e1-8348-dc6df37f0287" containerName="registry-server"
Jul 22 01:51:20.036788 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.036658    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4354f739-ab3a-42e1-8348-dc6df37f0287" containerName="registry-server"
Jul 22 01:51:20.039860 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.039839    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.042455 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.042432    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l7cfb\""
Jul 22 01:51:20.049809 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.049788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"]
Jul 22 01:51:20.149448 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.149412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cz7\" (UniqueName: \"kubernetes.io/projected/dbfa60fd-edca-4363-943b-cae0b3550d5c-kube-api-access-n9cz7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.149619 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.149479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbfa60fd-edca-4363-943b-cae0b3550d5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.250506 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.250471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbfa60fd-edca-4363-943b-cae0b3550d5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.250669 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.250520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cz7\" (UniqueName: \"kubernetes.io/projected/dbfa60fd-edca-4363-943b-cae0b3550d5c-kube-api-access-n9cz7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.250873 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.250853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbfa60fd-edca-4363-943b-cae0b3550d5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.259200 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.259178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cz7\" (UniqueName: \"kubernetes.io/projected/dbfa60fd-edca-4363-943b-cae0b3550d5c-kube-api-access-n9cz7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.350002 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.349910    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:20.477689 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.477666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"]
Jul 22 01:51:20.480236 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:20.480210    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfa60fd_edca_4363_943b_cae0b3550d5c.slice/crio-875e9fd7b61ed091d04c28b75da6adc365127f55a37599cef5b63ba02073204b WatchSource:0}: Error finding container 875e9fd7b61ed091d04c28b75da6adc365127f55a37599cef5b63ba02073204b: Status 404 returned error can't find the container with id 875e9fd7b61ed091d04c28b75da6adc365127f55a37599cef5b63ba02073204b
Jul 22 01:51:20.589997 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:20.589972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt" event={"ID":"dbfa60fd-edca-4363-943b-cae0b3550d5c","Type":"ContainerStarted","Data":"875e9fd7b61ed091d04c28b75da6adc365127f55a37599cef5b63ba02073204b"}
Jul 22 01:51:25.277202 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.277167    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-j484c"]
Jul 22 01:51:25.283342 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.283320    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:25.287284 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.287259    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rq4pk\""
Jul 22 01:51:25.294122 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.294100    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-j484c"]
Jul 22 01:51:25.392343 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.392311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljjq\" (UniqueName: \"kubernetes.io/projected/f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e-kube-api-access-8ljjq\") pod \"authorino-operator-75cc8d5cb7-j484c\" (UID: \"f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:25.493805 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.493770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljjq\" (UniqueName: \"kubernetes.io/projected/f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e-kube-api-access-8ljjq\") pod \"authorino-operator-75cc8d5cb7-j484c\" (UID: \"f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:25.505484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.505456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljjq\" (UniqueName: \"kubernetes.io/projected/f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e-kube-api-access-8ljjq\") pod \"authorino-operator-75cc8d5cb7-j484c\" (UID: \"f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:25.593321 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:25.593237    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:26.603310 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:26.603290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-j484c"]
Jul 22 01:51:26.605902 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:26.605870    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f2bb8c_16eb_4bcf_9e01_8536ad6a803e.slice/crio-227cd75e7f3da6d500b4a2657a74fd751e683274ecb74855a96ecc04c4332e06 WatchSource:0}: Error finding container 227cd75e7f3da6d500b4a2657a74fd751e683274ecb74855a96ecc04c4332e06: Status 404 returned error can't find the container with id 227cd75e7f3da6d500b4a2657a74fd751e683274ecb74855a96ecc04c4332e06
Jul 22 01:51:26.618086 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:26.618057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c" event={"ID":"f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e","Type":"ContainerStarted","Data":"227cd75e7f3da6d500b4a2657a74fd751e683274ecb74855a96ecc04c4332e06"}
Jul 22 01:51:27.631365 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:27.631114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt" event={"ID":"dbfa60fd-edca-4363-943b-cae0b3550d5c","Type":"ContainerStarted","Data":"f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0"}
Jul 22 01:51:27.631365 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:27.631257    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:27.678281 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:27.678228    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt" podStartSLOduration=1.6180927010000001 podStartE2EDuration="7.678209779s" totalImagesPullingTime="6.060117078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:27.67652677 +0000 UTC m=+399.179385001" watchObservedRunningTime="2026-07-22 01:51:27.678209779 +0000 UTC m=+399.181068009"
Jul 22 01:51:29.224991 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.224960    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"]
Jul 22 01:51:29.228100 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.228080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:29.230933 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.230909    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xc7dl\""
Jul 22 01:51:29.231027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.230982    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 22 01:51:29.238476 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.238456    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"]
Jul 22 01:51:29.328075 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.328003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvwt\" (UniqueName: \"kubernetes.io/projected/a45c1928-0e71-42c8-be8a-c3219dba4c4e-kube-api-access-gqvwt\") pod \"dns-operator-controller-manager-864dd78c4c-nf65f\" (UID: \"a45c1928-0e71-42c8-be8a-c3219dba4c4e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:29.429240 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.429204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvwt\" (UniqueName: \"kubernetes.io/projected/a45c1928-0e71-42c8-be8a-c3219dba4c4e-kube-api-access-gqvwt\") pod \"dns-operator-controller-manager-864dd78c4c-nf65f\" (UID: \"a45c1928-0e71-42c8-be8a-c3219dba4c4e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:29.437541 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.437518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvwt\" (UniqueName: \"kubernetes.io/projected/a45c1928-0e71-42c8-be8a-c3219dba4c4e-kube-api-access-gqvwt\") pod \"dns-operator-controller-manager-864dd78c4c-nf65f\" (UID: \"a45c1928-0e71-42c8-be8a-c3219dba4c4e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:29.539170 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.539136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:29.598634 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.598603    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"]
Jul 22 01:51:29.603289 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.603193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.607224 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.606772    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 22 01:51:29.607224 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.607144    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 22 01:51:29.608471 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.608449    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-clbcz\""
Jul 22 01:51:29.610861 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.610820    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"]
Jul 22 01:51:29.642630 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.642604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c" event={"ID":"f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e","Type":"ContainerStarted","Data":"090f907719ee8732b8adc0a107dc17f1bf690d0fdddfee22e138b487eb401f74"}
Jul 22 01:51:29.642777 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.642754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:29.663622 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.663574    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c" podStartSLOduration=2.242173368 podStartE2EDuration="4.663558636s" totalImagesPullingTime="2.421385268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:29.662564115 +0000 UTC m=+401.165422346" watchObservedRunningTime="2026-07-22 01:51:29.663558636 +0000 UTC m=+401.166416938"
Jul 22 01:51:29.682265 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.682245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"]
Jul 22 01:51:29.684447 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:29.684421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45c1928_0e71_42c8_be8a_c3219dba4c4e.slice/crio-69fa691c34bb7149784a142a98a6298a9a9ec2ea81cf7b3d694d64f1af770a08 WatchSource:0}: Error finding container 69fa691c34bb7149784a142a98a6298a9a9ec2ea81cf7b3d694d64f1af770a08: Status 404 returned error can't find the container with id 69fa691c34bb7149784a142a98a6298a9a9ec2ea81cf7b3d694d64f1af770a08
Jul 22 01:51:29.731289 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.731266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af95db8b-b50f-40ab-b971-a779024a53d0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.731416 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.731297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af95db8b-b50f-40ab-b971-a779024a53d0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.731416 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.731345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plhr\" (UniqueName: \"kubernetes.io/projected/af95db8b-b50f-40ab-b971-a779024a53d0-kube-api-access-5plhr\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.832638 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.832607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af95db8b-b50f-40ab-b971-a779024a53d0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.832638 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.832641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plhr\" (UniqueName: \"kubernetes.io/projected/af95db8b-b50f-40ab-b971-a779024a53d0-kube-api-access-5plhr\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.832889 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.832695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af95db8b-b50f-40ab-b971-a779024a53d0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.832889 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:29.832800    2561 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 22 01:51:29.832889 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:29.832851    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af95db8b-b50f-40ab-b971-a779024a53d0-plugin-serving-cert podName:af95db8b-b50f-40ab-b971-a779024a53d0 nodeName:}" failed. No retries permitted until 2026-07-22 01:51:30.332835477 +0000 UTC m=+401.835693683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/af95db8b-b50f-40ab-b971-a779024a53d0-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-7tsfk" (UID: "af95db8b-b50f-40ab-b971-a779024a53d0") : secret "plugin-serving-cert" not found
Jul 22 01:51:29.833317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.833295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af95db8b-b50f-40ab-b971-a779024a53d0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:29.844329 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:29.844273    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plhr\" (UniqueName: \"kubernetes.io/projected/af95db8b-b50f-40ab-b971-a779024a53d0-kube-api-access-5plhr\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:30.337630 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:30.337592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af95db8b-b50f-40ab-b971-a779024a53d0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:30.340017 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:30.339994    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af95db8b-b50f-40ab-b971-a779024a53d0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7tsfk\" (UID: \"af95db8b-b50f-40ab-b971-a779024a53d0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:30.516698 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:30.516665    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"
Jul 22 01:51:30.649633 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:30.649542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f" event={"ID":"a45c1928-0e71-42c8-be8a-c3219dba4c4e","Type":"ContainerStarted","Data":"69fa691c34bb7149784a142a98a6298a9a9ec2ea81cf7b3d694d64f1af770a08"}
Jul 22 01:51:30.841787 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:30.841762    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk"]
Jul 22 01:51:30.855675 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:30.855643    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf95db8b_b50f_40ab_b971_a779024a53d0.slice/crio-13f401d3bcb1e48c8cc9fc5ed0b0cdcae9536e18b9c03e928f44f0d43fc7b171 WatchSource:0}: Error finding container 13f401d3bcb1e48c8cc9fc5ed0b0cdcae9536e18b9c03e928f44f0d43fc7b171: Status 404 returned error can't find the container with id 13f401d3bcb1e48c8cc9fc5ed0b0cdcae9536e18b9c03e928f44f0d43fc7b171
Jul 22 01:51:31.656071 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:31.656031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk" event={"ID":"af95db8b-b50f-40ab-b971-a779024a53d0","Type":"ContainerStarted","Data":"13f401d3bcb1e48c8cc9fc5ed0b0cdcae9536e18b9c03e928f44f0d43fc7b171"}
Jul 22 01:51:31.658164 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:31.658130    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f" event={"ID":"a45c1928-0e71-42c8-be8a-c3219dba4c4e","Type":"ContainerStarted","Data":"26ba02e9d9c6041953cf9a94d4829b0ca6dc67f2d2ef52585efd213cd32270ec"}
Jul 22 01:51:31.658306 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:31.658288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:31.677159 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:31.677079    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f" podStartSLOduration=0.905156533 podStartE2EDuration="2.677063804s" totalImagesPullingTime="1.771907271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:31.67464881 +0000 UTC m=+403.177507038" watchObservedRunningTime="2026-07-22 01:51:31.677063804 +0000 UTC m=+403.179922033"
Jul 22 01:51:32.689232 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.689200    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"]
Jul 22 01:51:32.692931 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.692909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:32.695328 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.695305    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5tcjw\""
Jul 22 01:51:32.703315 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.703292    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"]
Jul 22 01:51:32.861082 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.861039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrb8\" (UniqueName: \"kubernetes.io/projected/8a6bf4f4-716e-48ac-9329-9e0e216825c9-kube-api-access-jfrb8\") pod \"limitador-operator-controller-manager-859d7b78bd-jdr98\" (UID: \"8a6bf4f4-716e-48ac-9329-9e0e216825c9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:32.961644 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.961561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrb8\" (UniqueName: \"kubernetes.io/projected/8a6bf4f4-716e-48ac-9329-9e0e216825c9-kube-api-access-jfrb8\") pod \"limitador-operator-controller-manager-859d7b78bd-jdr98\" (UID: \"8a6bf4f4-716e-48ac-9329-9e0e216825c9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:32.972324 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:32.972296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrb8\" (UniqueName: \"kubernetes.io/projected/8a6bf4f4-716e-48ac-9329-9e0e216825c9-kube-api-access-jfrb8\") pod \"limitador-operator-controller-manager-859d7b78bd-jdr98\" (UID: \"8a6bf4f4-716e-48ac-9329-9e0e216825c9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:33.006476 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:33.006387    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:33.170937 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:33.170858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"]
Jul 22 01:51:33.173578 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:33.173521    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6bf4f4_716e_48ac_9329_9e0e216825c9.slice/crio-fbb839da580712e24a2b136107724b9d05608b4d69df3fe6b3f2cd2e1145df79 WatchSource:0}: Error finding container fbb839da580712e24a2b136107724b9d05608b4d69df3fe6b3f2cd2e1145df79: Status 404 returned error can't find the container with id fbb839da580712e24a2b136107724b9d05608b4d69df3fe6b3f2cd2e1145df79
Jul 22 01:51:33.670315 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:33.670280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98" event={"ID":"8a6bf4f4-716e-48ac-9329-9e0e216825c9","Type":"ContainerStarted","Data":"fbb839da580712e24a2b136107724b9d05608b4d69df3fe6b3f2cd2e1145df79"}
Jul 22 01:51:36.688882 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:36.688769    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98" event={"ID":"8a6bf4f4-716e-48ac-9329-9e0e216825c9","Type":"ContainerStarted","Data":"f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb"}
Jul 22 01:51:36.689256 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:36.688876    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:36.707408 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:36.707351    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98" podStartSLOduration=1.492115122 podStartE2EDuration="4.707336498s" totalImagesPullingTime="3.215221376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:36.706018867 +0000 UTC m=+408.208877097" watchObservedRunningTime="2026-07-22 01:51:36.707336498 +0000 UTC m=+408.210194738"
Jul 22 01:51:38.639226 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:38.639197    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:40.548231 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.548194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"]
Jul 22 01:51:40.548640 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.548479    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" containerName="manager" containerID="cri-o://f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0" gracePeriod=2
Jul 22 01:51:40.564928 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.564897    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"]
Jul 22 01:51:40.580484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.579872    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"]
Jul 22 01:51:40.580484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.580299    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" containerName="manager"
Jul 22 01:51:40.580484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.580315    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" containerName="manager"
Jul 22 01:51:40.580484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.580401    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" containerName="manager"
Jul 22 01:51:40.586413 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.585852    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"]
Jul 22 01:51:40.586413 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.586031    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" containerName="manager" containerID="cri-o://f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb" gracePeriod=2
Jul 22 01:51:40.586413 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.586154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.587498 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.587475    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"]
Jul 22 01:51:40.604055 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.603997    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"]
Jul 22 01:51:40.613064 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.613040    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"]
Jul 22 01:51:40.613460 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.613441    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" containerName="manager"
Jul 22 01:51:40.613460 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.613460    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" containerName="manager"
Jul 22 01:51:40.613620 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.613544    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" containerName="manager"
Jul 22 01:51:40.616766 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.616745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:51:40.620623 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:40.620579    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:40.622929 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:40.622898    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:40.625148 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:40.625118    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:40.626221 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.626200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"]
Jul 22 01:51:40.647315 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:40.647281    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:40.652073 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.652050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-j484c"
Jul 22 01:51:40.673196 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:40.673163    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:40.675127 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:40.675099    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:40.728872 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.728842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9788db2-79d6-438e-b25d-ea3da85af1b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rjtff\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.729048 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.728888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvrl\" (UniqueName: \"kubernetes.io/projected/8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c-kube-api-access-njvrl\") pod \"limitador-operator-controller-manager-859d7b78bd-wsdgd\" (UID: \"8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:51:40.729048 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.728968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrdw\" (UniqueName: \"kubernetes.io/projected/e9788db2-79d6-438e-b25d-ea3da85af1b0-kube-api-access-bfrdw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rjtff\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.830142 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.830044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9788db2-79d6-438e-b25d-ea3da85af1b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rjtff\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.830142 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.830103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvrl\" (UniqueName: \"kubernetes.io/projected/8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c-kube-api-access-njvrl\") pod \"limitador-operator-controller-manager-859d7b78bd-wsdgd\" (UID: \"8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:51:40.830389 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.830171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrdw\" (UniqueName: \"kubernetes.io/projected/e9788db2-79d6-438e-b25d-ea3da85af1b0-kube-api-access-bfrdw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rjtff\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.830521 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.830497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9788db2-79d6-438e-b25d-ea3da85af1b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rjtff\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.841545 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.841520    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvrl\" (UniqueName: \"kubernetes.io/projected/8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c-kube-api-access-njvrl\") pod \"limitador-operator-controller-manager-859d7b78bd-wsdgd\" (UID: \"8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:51:40.841681 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.841670    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrdw\" (UniqueName: \"kubernetes.io/projected/e9788db2-79d6-438e-b25d-ea3da85af1b0-kube-api-access-bfrdw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rjtff\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.975614 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.975575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:40.983069 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:40.983050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:51:42.667160 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:42.667129    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nf65f"
Jul 22 01:51:48.954563 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:48.954531    2561 scope.go:117] "RemoveContainer" containerID="db35b7cd353e906649339fb97d4607da45aca3092debf9eb40a7f621eaf78cfc"
Jul 22 01:51:51.983147 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:51.982979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:51.983147 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:51.983078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"]
Jul 22 01:51:51.984222 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:51.984195    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9788db2_79d6_438e_b25d_ea3da85af1b0.slice/crio-86e59810c5f073745c51572b4fb013bd29bfd4a194c2cda4c0abdc14b1899f26 WatchSource:0}: Error finding container 86e59810c5f073745c51572b4fb013bd29bfd4a194c2cda4c0abdc14b1899f26: Status 404 returned error can't find the container with id 86e59810c5f073745c51572b4fb013bd29bfd4a194c2cda4c0abdc14b1899f26
Jul 22 01:51:51.985166 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:51.985103    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:51.989379 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:51.989359    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:51.991430 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:51.991338    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:51.993261 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:51.993236    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:52.003496 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.003476    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"]
Jul 22 01:51:52.006286 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:52.006255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3357b1_4e3b_4d8f_a03f_d2e3c4c61f9c.slice/crio-c903d328114732ca565255cb02a498879fd7e29a63d61b5aa39b549a004b18f2 WatchSource:0}: Error finding container c903d328114732ca565255cb02a498879fd7e29a63d61b5aa39b549a004b18f2: Status 404 returned error can't find the container with id c903d328114732ca565255cb02a498879fd7e29a63d61b5aa39b549a004b18f2
Jul 22 01:51:52.124439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.124364    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbfa60fd-edca-4363-943b-cae0b3550d5c-extensions-socket-volume\") pod \"dbfa60fd-edca-4363-943b-cae0b3550d5c\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") "
Jul 22 01:51:52.124439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.124419    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9cz7\" (UniqueName: \"kubernetes.io/projected/dbfa60fd-edca-4363-943b-cae0b3550d5c-kube-api-access-n9cz7\") pod \"dbfa60fd-edca-4363-943b-cae0b3550d5c\" (UID: \"dbfa60fd-edca-4363-943b-cae0b3550d5c\") "
Jul 22 01:51:52.124589 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.124458    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfrb8\" (UniqueName: \"kubernetes.io/projected/8a6bf4f4-716e-48ac-9329-9e0e216825c9-kube-api-access-jfrb8\") pod \"8a6bf4f4-716e-48ac-9329-9e0e216825c9\" (UID: \"8a6bf4f4-716e-48ac-9329-9e0e216825c9\") "
Jul 22 01:51:52.124956 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.124932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfa60fd-edca-4363-943b-cae0b3550d5c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dbfa60fd-edca-4363-943b-cae0b3550d5c" (UID: "dbfa60fd-edca-4363-943b-cae0b3550d5c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:51:52.126465 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.126441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfa60fd-edca-4363-943b-cae0b3550d5c-kube-api-access-n9cz7" (OuterVolumeSpecName: "kube-api-access-n9cz7") pod "dbfa60fd-edca-4363-943b-cae0b3550d5c" (UID: "dbfa60fd-edca-4363-943b-cae0b3550d5c"). InnerVolumeSpecName "kube-api-access-n9cz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:51:52.126465 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.126450    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6bf4f4-716e-48ac-9329-9e0e216825c9-kube-api-access-jfrb8" (OuterVolumeSpecName: "kube-api-access-jfrb8") pod "8a6bf4f4-716e-48ac-9329-9e0e216825c9" (UID: "8a6bf4f4-716e-48ac-9329-9e0e216825c9"). InnerVolumeSpecName "kube-api-access-jfrb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:51:52.225035 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.225004    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbfa60fd-edca-4363-943b-cae0b3550d5c-extensions-socket-volume\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:51:52.225035 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.225029    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9cz7\" (UniqueName: \"kubernetes.io/projected/dbfa60fd-edca-4363-943b-cae0b3550d5c-kube-api-access-n9cz7\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:51:52.225035 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.225040    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrb8\" (UniqueName: \"kubernetes.io/projected/8a6bf4f4-716e-48ac-9329-9e0e216825c9-kube-api-access-jfrb8\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:51:52.769125 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.769091    2561 generic.go:356] "Generic (PLEG): container finished" podID="dbfa60fd-edca-4363-943b-cae0b3550d5c" containerID="f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0" exitCode=0
Jul 22 01:51:52.769289 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.769139    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:52.769289 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.769144    2561 scope.go:117] "RemoveContainer" containerID="f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0"
Jul 22 01:51:52.770899 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.770873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd" event={"ID":"8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c","Type":"ContainerStarted","Data":"bfc484ae2c4ad9df911011a6372e182b5a0cb0cdedfa9d9316c757b5f8ecc0dd"}
Jul 22 01:51:52.771027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.770917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd" event={"ID":"8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c","Type":"ContainerStarted","Data":"c903d328114732ca565255cb02a498879fd7e29a63d61b5aa39b549a004b18f2"}
Jul 22 01:51:52.771027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.770975    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:51:52.771523 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.771494    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:52.772582 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.772544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk" event={"ID":"af95db8b-b50f-40ab-b971-a779024a53d0","Type":"ContainerStarted","Data":"96c67390eb059d81a79a65aceeb5d38704a6e3bdb74f5cafe60e0c48d444b36f"}
Jul 22 01:51:52.773440 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.773412    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:52.774278 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.774257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff" event={"ID":"e9788db2-79d6-438e-b25d-ea3da85af1b0","Type":"ContainerStarted","Data":"b0b5868d9ddf94dd1db45965788e975ee789cb23a335638dace14737612982bb"}
Jul 22 01:51:52.774375 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.774299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff" event={"ID":"e9788db2-79d6-438e-b25d-ea3da85af1b0","Type":"ContainerStarted","Data":"86e59810c5f073745c51572b4fb013bd29bfd4a194c2cda4c0abdc14b1899f26"}
Jul 22 01:51:52.774375 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.774351    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:52.775564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.775546    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" containerID="f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb" exitCode=0
Jul 22 01:51:52.775643 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.775581    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:52.778241 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.778144    2561 scope.go:117] "RemoveContainer" containerID="f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0"
Jul 22 01:51:52.778444 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.778418    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0\": container with ID starting with f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0 not found: ID does not exist" containerID="f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0"
Jul 22 01:51:52.778553 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.778451    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0"} err="failed to get container status \"f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0\": rpc error: code = NotFound desc = could not find container \"f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0\": container with ID starting with f451e92afef913c66acbfaa6cbb1aa293dd87423a796224b35c9bc2f02fa75a0 not found: ID does not exist"
Jul 22 01:51:52.778553 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.778474    2561 scope.go:117] "RemoveContainer" containerID="f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb"
Jul 22 01:51:52.785830 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.785812    2561 scope.go:117] "RemoveContainer" containerID="f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb"
Jul 22 01:51:52.786065 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.786049    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb\": container with ID starting with f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb not found: ID does not exist" containerID="f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb"
Jul 22 01:51:52.786111 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.786071    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb"} err="failed to get container status \"f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb\": rpc error: code = NotFound desc = could not find container \"f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb\": container with ID starting with f61bfe4603a881893a7539f64dc3c54621d83b25622a66379aa6f1cf62d23edb not found: ID does not exist"
Jul 22 01:51:52.790323 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.790300    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:52.811543 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.811503    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd" podStartSLOduration=12.811490507 podStartE2EDuration="12.811490507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:52.810602468 +0000 UTC m=+424.313460701" watchObservedRunningTime="2026-07-22 01:51:52.811490507 +0000 UTC m=+424.314348713"
Jul 22 01:51:52.811792 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.811763    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7tsfk" podStartSLOduration=2.753221258 podStartE2EDuration="23.811754288s" totalImagesPullingTime="21.05853303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:52.788269517 +0000 UTC m=+424.291127747" watchObservedRunningTime="2026-07-22 01:51:52.811754288 +0000 UTC m=+424.314612507"
Jul 22 01:51:52.812682 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.812659    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:52.834439 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:52.834401    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff" podStartSLOduration=12.834390016 podStartE2EDuration="12.834390016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:52.833641802 +0000 UTC m=+424.336500032" watchObservedRunningTime="2026-07-22 01:51:52.834390016 +0000 UTC m=+424.337248246"
Jul 22 01:51:52.836855 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.836830    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jdr98\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jdr98"
Jul 22 01:51:52.838829 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:51:52.838798    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nfgdt\" is forbidden: User \"system:node:ip-10-0-130-189.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-189.ec2.internal' and this object" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfgdt"
Jul 22 01:51:53.106183 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:53.106101    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6bf4f4-716e-48ac-9329-9e0e216825c9" path="/var/lib/kubelet/pods/8a6bf4f4-716e-48ac-9329-9e0e216825c9/volumes"
Jul 22 01:51:53.106621 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:53.106586    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbfa60fd-edca-4363-943b-cae0b3550d5c" path="/var/lib/kubelet/pods/dbfa60fd-edca-4363-943b-cae0b3550d5c/volumes"
Jul 22 01:51:56.678807 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.678763    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"]
Jul 22 01:51:56.679455 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.678988    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff" podUID="e9788db2-79d6-438e-b25d-ea3da85af1b0" containerName="manager" containerID="cri-o://b0b5868d9ddf94dd1db45965788e975ee789cb23a335638dace14737612982bb" gracePeriod=10
Jul 22 01:51:56.798695 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.798667    2561 generic.go:356] "Generic (PLEG): container finished" podID="e9788db2-79d6-438e-b25d-ea3da85af1b0" containerID="b0b5868d9ddf94dd1db45965788e975ee789cb23a335638dace14737612982bb" exitCode=0
Jul 22 01:51:56.798806 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.798717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff" event={"ID":"e9788db2-79d6-438e-b25d-ea3da85af1b0","Type":"ContainerDied","Data":"b0b5868d9ddf94dd1db45965788e975ee789cb23a335638dace14737612982bb"}
Jul 22 01:51:56.927251 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.927232    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:56.959536 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.959474    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"]
Jul 22 01:51:56.959855 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.959836    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9788db2-79d6-438e-b25d-ea3da85af1b0" containerName="manager"
Jul 22 01:51:56.959855 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.959856    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9788db2-79d6-438e-b25d-ea3da85af1b0" containerName="manager"
Jul 22 01:51:56.960063 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.959945    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9788db2-79d6-438e-b25d-ea3da85af1b0" containerName="manager"
Jul 22 01:51:56.962313 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.962293    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfrdw\" (UniqueName: \"kubernetes.io/projected/e9788db2-79d6-438e-b25d-ea3da85af1b0-kube-api-access-bfrdw\") pod \"e9788db2-79d6-438e-b25d-ea3da85af1b0\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") "
Jul 22 01:51:56.962407 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.962344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9788db2-79d6-438e-b25d-ea3da85af1b0-extensions-socket-volume\") pod \"e9788db2-79d6-438e-b25d-ea3da85af1b0\" (UID: \"e9788db2-79d6-438e-b25d-ea3da85af1b0\") "
Jul 22 01:51:56.962745 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.962704    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9788db2-79d6-438e-b25d-ea3da85af1b0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e9788db2-79d6-438e-b25d-ea3da85af1b0" (UID: "e9788db2-79d6-438e-b25d-ea3da85af1b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 01:51:56.962829 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.962812    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:56.964353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.964330    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9788db2-79d6-438e-b25d-ea3da85af1b0-kube-api-access-bfrdw" (OuterVolumeSpecName: "kube-api-access-bfrdw") pod "e9788db2-79d6-438e-b25d-ea3da85af1b0" (UID: "e9788db2-79d6-438e-b25d-ea3da85af1b0"). InnerVolumeSpecName "kube-api-access-bfrdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:51:56.978816 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:56.978790    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"]
Jul 22 01:51:57.063182 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.063145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccq6s\" (UniqueName: \"kubernetes.io/projected/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-kube-api-access-ccq6s\") pod \"kuadrant-operator-controller-manager-86878c769f-5g7b4\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.063369 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.063193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5g7b4\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.063369 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.063269    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9788db2-79d6-438e-b25d-ea3da85af1b0-extensions-socket-volume\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:51:57.063369 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.063294    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfrdw\" (UniqueName: \"kubernetes.io/projected/e9788db2-79d6-438e-b25d-ea3da85af1b0-kube-api-access-bfrdw\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:51:57.164218 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.164179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccq6s\" (UniqueName: \"kubernetes.io/projected/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-kube-api-access-ccq6s\") pod \"kuadrant-operator-controller-manager-86878c769f-5g7b4\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.164472 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.164444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5g7b4\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.164826 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.164803    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5g7b4\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.175353 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.175327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccq6s\" (UniqueName: \"kubernetes.io/projected/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-kube-api-access-ccq6s\") pod \"kuadrant-operator-controller-manager-86878c769f-5g7b4\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.278848 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.278826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.408867 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.408837    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"]
Jul 22 01:51:57.411363 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:51:57.411339    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5abf00_0010_40d6_be44_dbd3d3d24f8c.slice/crio-4803baa4a990ce1c3d1eb2a9740799431770a8323c2b0deb43193530ed0abd40 WatchSource:0}: Error finding container 4803baa4a990ce1c3d1eb2a9740799431770a8323c2b0deb43193530ed0abd40: Status 404 returned error can't find the container with id 4803baa4a990ce1c3d1eb2a9740799431770a8323c2b0deb43193530ed0abd40
Jul 22 01:51:57.809603 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.809577    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"
Jul 22 01:51:57.810086 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.809570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff" event={"ID":"e9788db2-79d6-438e-b25d-ea3da85af1b0","Type":"ContainerDied","Data":"86e59810c5f073745c51572b4fb013bd29bfd4a194c2cda4c0abdc14b1899f26"}
Jul 22 01:51:57.810603 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.809766    2561 scope.go:117] "RemoveContainer" containerID="b0b5868d9ddf94dd1db45965788e975ee789cb23a335638dace14737612982bb"
Jul 22 01:51:57.811918 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.811455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4" event={"ID":"aa5abf00-0010-40d6-be44-dbd3d3d24f8c","Type":"ContainerStarted","Data":"523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38"}
Jul 22 01:51:57.811918 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.811498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4" event={"ID":"aa5abf00-0010-40d6-be44-dbd3d3d24f8c","Type":"ContainerStarted","Data":"4803baa4a990ce1c3d1eb2a9740799431770a8323c2b0deb43193530ed0abd40"}
Jul 22 01:51:57.811918 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.811542    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:51:57.825593 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.825575    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"]
Jul 22 01:51:57.831389 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.831366    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rjtff"]
Jul 22 01:51:57.853957 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:57.853912    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4" podStartSLOduration=1.853900244 podStartE2EDuration="1.853900244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:51:57.852618442 +0000 UTC m=+429.355476670" watchObservedRunningTime="2026-07-22 01:51:57.853900244 +0000 UTC m=+429.356758473"
Jul 22 01:51:59.102007 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:51:59.101972    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9788db2-79d6-438e-b25d-ea3da85af1b0" path="/var/lib/kubelet/pods/e9788db2-79d6-438e-b25d-ea3da85af1b0/volumes"
Jul 22 01:52:03.784610 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:03.784579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wsdgd"
Jul 22 01:52:08.819930 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:08.819902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 01:52:29.444674 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.444640    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:29.490540 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.490514    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:29.490687 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.490606    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.493019 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.492993    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 22 01:52:29.539230 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.539196    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:29.618602 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.618564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-config-file\") pod \"limitador-limitador-64c94fd64f-bpn6n\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.618797 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.618635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46sxl\" (UniqueName: \"kubernetes.io/projected/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-kube-api-access-46sxl\") pod \"limitador-limitador-64c94fd64f-bpn6n\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.719403 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.719329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-config-file\") pod \"limitador-limitador-64c94fd64f-bpn6n\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.719403 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.719375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46sxl\" (UniqueName: \"kubernetes.io/projected/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-kube-api-access-46sxl\") pod \"limitador-limitador-64c94fd64f-bpn6n\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.720024 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.719996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-config-file\") pod \"limitador-limitador-64c94fd64f-bpn6n\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.727205 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.727186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46sxl\" (UniqueName: \"kubernetes.io/projected/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-kube-api-access-46sxl\") pod \"limitador-limitador-64c94fd64f-bpn6n\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.800629 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.800599    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:29.933886 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.933791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:29.936252 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:52:29.936229    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191c93e9_8a04_46f6_b1c1_5c3b2d87f40d.slice/crio-f9d2cd443968f9232a73c692f904c66dd211998c125a929c87b6ab667c45542f WatchSource:0}: Error finding container f9d2cd443968f9232a73c692f904c66dd211998c125a929c87b6ab667c45542f: Status 404 returned error can't find the container with id f9d2cd443968f9232a73c692f904c66dd211998c125a929c87b6ab667c45542f
Jul 22 01:52:29.951047 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:29.951023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" event={"ID":"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d","Type":"ContainerStarted","Data":"f9d2cd443968f9232a73c692f904c66dd211998c125a929c87b6ab667c45542f"}
Jul 22 01:52:30.262058 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.262026    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-g9nhc"]
Jul 22 01:52:30.266748 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.266713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:30.269116 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.269098    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-24slz\""
Jul 22 01:52:30.276370 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.276350    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9nhc"]
Jul 22 01:52:30.323457 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.323433    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mv5\" (UniqueName: \"kubernetes.io/projected/a9a43983-2395-4bd9-848e-2d5f80d0ffb6-kube-api-access-c4mv5\") pod \"authorino-58cd955675-g9nhc\" (UID: \"a9a43983-2395-4bd9-848e-2d5f80d0ffb6\") " pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:30.424780 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.424719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mv5\" (UniqueName: \"kubernetes.io/projected/a9a43983-2395-4bd9-848e-2d5f80d0ffb6-kube-api-access-c4mv5\") pod \"authorino-58cd955675-g9nhc\" (UID: \"a9a43983-2395-4bd9-848e-2d5f80d0ffb6\") " pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:30.433024 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.432999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mv5\" (UniqueName: \"kubernetes.io/projected/a9a43983-2395-4bd9-848e-2d5f80d0ffb6-kube-api-access-c4mv5\") pod \"authorino-58cd955675-g9nhc\" (UID: \"a9a43983-2395-4bd9-848e-2d5f80d0ffb6\") " pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:30.576466 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.576399    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:30.720593 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.720556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9nhc"]
Jul 22 01:52:30.728171 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:52:30.728140    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9a43983_2395_4bd9_848e_2d5f80d0ffb6.slice/crio-90ee726bc4e20dad56ac0287affaefd5b89d872ce98d691c68fe10b8291a2e96 WatchSource:0}: Error finding container 90ee726bc4e20dad56ac0287affaefd5b89d872ce98d691c68fe10b8291a2e96: Status 404 returned error can't find the container with id 90ee726bc4e20dad56ac0287affaefd5b89d872ce98d691c68fe10b8291a2e96
Jul 22 01:52:30.960169 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:30.960086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9nhc" event={"ID":"a9a43983-2395-4bd9-848e-2d5f80d0ffb6","Type":"ContainerStarted","Data":"90ee726bc4e20dad56ac0287affaefd5b89d872ce98d691c68fe10b8291a2e96"}
Jul 22 01:52:32.665967 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:52:32.665922    2561 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:e9c3637d213489c047e3af9a1c76b1d453c048341032e79f9ae4a3336c176898: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/kuadrant/limitador:v2.3.0"
Jul 22 01:52:32.666378 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:52:32.666051    2561 kuberuntime_manager.go:1459] "Unhandled Error" err="container limitador start failed in pod limitador-limitador-64c94fd64f-bpn6n_kuadrant-system(191c93e9-8a04-46f6-b1c1-5c3b2d87f40d): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:e9c3637d213489c047e3af9a1c76b1d453c048341032e79f9ae4a3336c176898: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 22 01:52:32.666378 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:52:32.666114    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"limitador\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:e9c3637d213489c047e3af9a1c76b1d453c048341032e79f9ae4a3336c176898: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d"
Jul 22 01:52:32.973353 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:52:32.973317    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"limitador\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/kuadrant/limitador:v2.3.0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:e9c3637d213489c047e3af9a1c76b1d453c048341032e79f9ae4a3336c176898: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d"
Jul 22 01:52:34.984546 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:34.984515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9nhc" event={"ID":"a9a43983-2395-4bd9-848e-2d5f80d0ffb6","Type":"ContainerStarted","Data":"a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267"}
Jul 22 01:52:35.000417 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:35.000371    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-g9nhc" podStartSLOduration=1.635779857 podStartE2EDuration="5.000354033s" totalImagesPullingTime="3.364574176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:52:34.99990366 +0000 UTC m=+466.502761890" watchObservedRunningTime="2026-07-22 01:52:35.000354033 +0000 UTC m=+466.503212263"
Jul 22 01:52:35.374896 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:35.374866    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9nhc"]
Jul 22 01:52:36.992569 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:36.992532    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-g9nhc" podUID="a9a43983-2395-4bd9-848e-2d5f80d0ffb6" containerName="authorino" containerID="cri-o://a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267" gracePeriod=30
Jul 22 01:52:37.236968 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.236947    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:37.382272 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.382246    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4mv5\" (UniqueName: \"kubernetes.io/projected/a9a43983-2395-4bd9-848e-2d5f80d0ffb6-kube-api-access-c4mv5\") pod \"a9a43983-2395-4bd9-848e-2d5f80d0ffb6\" (UID: \"a9a43983-2395-4bd9-848e-2d5f80d0ffb6\") "
Jul 22 01:52:37.384317 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.384286    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a43983-2395-4bd9-848e-2d5f80d0ffb6-kube-api-access-c4mv5" (OuterVolumeSpecName: "kube-api-access-c4mv5") pod "a9a43983-2395-4bd9-848e-2d5f80d0ffb6" (UID: "a9a43983-2395-4bd9-848e-2d5f80d0ffb6"). InnerVolumeSpecName "kube-api-access-c4mv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:52:37.483671 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.483639    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4mv5\" (UniqueName: \"kubernetes.io/projected/a9a43983-2395-4bd9-848e-2d5f80d0ffb6-kube-api-access-c4mv5\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:52:37.998775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.998718    2561 generic.go:356] "Generic (PLEG): container finished" podID="a9a43983-2395-4bd9-848e-2d5f80d0ffb6" containerID="a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267" exitCode=0
Jul 22 01:52:37.998775 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.998759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9nhc" event={"ID":"a9a43983-2395-4bd9-848e-2d5f80d0ffb6","Type":"ContainerDied","Data":"a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267"}
Jul 22 01:52:37.999340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.998780    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g9nhc"
Jul 22 01:52:37.999340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.998806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g9nhc" event={"ID":"a9a43983-2395-4bd9-848e-2d5f80d0ffb6","Type":"ContainerDied","Data":"90ee726bc4e20dad56ac0287affaefd5b89d872ce98d691c68fe10b8291a2e96"}
Jul 22 01:52:37.999340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:37.998832    2561 scope.go:117] "RemoveContainer" containerID="a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267"
Jul 22 01:52:38.007558 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:38.007543    2561 scope.go:117] "RemoveContainer" containerID="a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267"
Jul 22 01:52:38.007817 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:52:38.007801    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267\": container with ID starting with a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267 not found: ID does not exist" containerID="a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267"
Jul 22 01:52:38.007910 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:38.007824    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267"} err="failed to get container status \"a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267\": rpc error: code = NotFound desc = could not find container \"a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267\": container with ID starting with a7cf781fc9066f1e82ffef28a29df1cd69596daa6a5e6183b953d1554a30a267 not found: ID does not exist"
Jul 22 01:52:38.024342 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:38.019380    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9nhc"]
Jul 22 01:52:38.024342 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:38.024297    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-g9nhc"]
Jul 22 01:52:39.101134 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:39.101102    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9a43983-2395-4bd9-848e-2d5f80d0ffb6" path="/var/lib/kubelet/pods/a9a43983-2395-4bd9-848e-2d5f80d0ffb6/volumes"
Jul 22 01:52:46.038311 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:46.038277    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" event={"ID":"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d","Type":"ContainerStarted","Data":"870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b"}
Jul 22 01:52:46.038722 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:46.038494    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:46.055595 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:46.055549    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" podStartSLOduration=1.8047687799999999 podStartE2EDuration="17.055534847s" totalImagesPullingTime="15.250766067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-22 01:52:46.054426462 +0000 UTC m=+477.557284692" watchObservedRunningTime="2026-07-22 01:52:46.055534847 +0000 UTC m=+477.558393142"
Jul 22 01:52:46.389456 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:46.389378    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:48.046853 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:48.046816    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" containerName="limitador" containerID="cri-o://870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b" gracePeriod=30
Jul 22 01:52:48.986292 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:48.986267    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:49.052605 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.052572    2561 generic.go:356] "Generic (PLEG): container finished" podID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" containerID="870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b" exitCode=0
Jul 22 01:52:49.053027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.052616    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"
Jul 22 01:52:49.053027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.052619    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" event={"ID":"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d","Type":"ContainerDied","Data":"870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b"}
Jul 22 01:52:49.053027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.052753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-bpn6n" event={"ID":"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d","Type":"ContainerDied","Data":"f9d2cd443968f9232a73c692f904c66dd211998c125a929c87b6ab667c45542f"}
Jul 22 01:52:49.053027 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.052788    2561 scope.go:117] "RemoveContainer" containerID="870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b"
Jul 22 01:52:49.060717 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.060701    2561 scope.go:117] "RemoveContainer" containerID="870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b"
Jul 22 01:52:49.060984 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:52:49.060967    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b\": container with ID starting with 870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b not found: ID does not exist" containerID="870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b"
Jul 22 01:52:49.061036 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.060992    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b"} err="failed to get container status \"870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b\": rpc error: code = NotFound desc = could not find container \"870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b\": container with ID starting with 870bf6ac0afa8facc0f30e096e3fed5e60b0132be1d232e8a7450bd7d9f6df8b not found: ID does not exist"
Jul 22 01:52:49.067218 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.067201    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46sxl\" (UniqueName: \"kubernetes.io/projected/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-kube-api-access-46sxl\") pod \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") "
Jul 22 01:52:49.067269 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.067257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-config-file\") pod \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\" (UID: \"191c93e9-8a04-46f6-b1c1-5c3b2d87f40d\") "
Jul 22 01:52:49.067564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.067544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-config-file" (OuterVolumeSpecName: "config-file") pod "191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" (UID: "191c93e9-8a04-46f6-b1c1-5c3b2d87f40d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 01:52:49.069098 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.069075    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-kube-api-access-46sxl" (OuterVolumeSpecName: "kube-api-access-46sxl") pod "191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" (UID: "191c93e9-8a04-46f6-b1c1-5c3b2d87f40d"). InnerVolumeSpecName "kube-api-access-46sxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:52:49.168574 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.168510    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46sxl\" (UniqueName: \"kubernetes.io/projected/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-kube-api-access-46sxl\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:52:49.168574 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.168533    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d-config-file\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:52:49.368603 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.368570    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:49.372311 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:49.372286    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-bpn6n"]
Jul 22 01:52:51.101807 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:51.101777    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" path="/var/lib/kubelet/pods/191c93e9-8a04-46f6-b1c1-5c3b2d87f40d/volumes"
Jul 22 01:52:52.163045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.162971    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"]
Jul 22 01:52:52.163370 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.163272    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a43983-2395-4bd9-848e-2d5f80d0ffb6" containerName="authorino"
Jul 22 01:52:52.163370 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.163287    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a43983-2395-4bd9-848e-2d5f80d0ffb6" containerName="authorino"
Jul 22 01:52:52.163370 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.163297    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" containerName="limitador"
Jul 22 01:52:52.163370 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.163302    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" containerName="limitador"
Jul 22 01:52:52.163370 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.163366    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191c93e9-8a04-46f6-b1c1-5c3b2d87f40d" containerName="limitador"
Jul 22 01:52:52.163515 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.163377    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a43983-2395-4bd9-848e-2d5f80d0ffb6" containerName="authorino"
Jul 22 01:52:52.167770 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.167749    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.173294 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.173272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-l9xrt\""
Jul 22 01:52:52.173513 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.173498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 22 01:52:52.173588 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.173500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 22 01:52:52.174295 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.174280    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 22 01:52:52.182169 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.182148    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"]
Jul 22 01:52:52.290273 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.290245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wgl\" (UniqueName: \"kubernetes.io/projected/2cd6b3e8-ed8b-4d34-9247-b07dad382d1f-kube-api-access-k5wgl\") pod \"postgres-69fb4594cf-7j62c\" (UID: \"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.290401 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.290286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2cd6b3e8-ed8b-4d34-9247-b07dad382d1f-data\") pod \"postgres-69fb4594cf-7j62c\" (UID: \"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.390838 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.390809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wgl\" (UniqueName: \"kubernetes.io/projected/2cd6b3e8-ed8b-4d34-9247-b07dad382d1f-kube-api-access-k5wgl\") pod \"postgres-69fb4594cf-7j62c\" (UID: \"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.391045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.390851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2cd6b3e8-ed8b-4d34-9247-b07dad382d1f-data\") pod \"postgres-69fb4594cf-7j62c\" (UID: \"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.391177 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.391162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2cd6b3e8-ed8b-4d34-9247-b07dad382d1f-data\") pod \"postgres-69fb4594cf-7j62c\" (UID: \"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.398946 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.398920    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wgl\" (UniqueName: \"kubernetes.io/projected/2cd6b3e8-ed8b-4d34-9247-b07dad382d1f-kube-api-access-k5wgl\") pod \"postgres-69fb4594cf-7j62c\" (UID: \"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.477780 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.477699    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:52.600878 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:52.600857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"]
Jul 22 01:52:52.602712 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:52:52.602679    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd6b3e8_ed8b_4d34_9247_b07dad382d1f.slice/crio-2626ab2a2ac683864c0963f9846908a154239e59df5d45fbb2096fac1d5dcb07 WatchSource:0}: Error finding container 2626ab2a2ac683864c0963f9846908a154239e59df5d45fbb2096fac1d5dcb07: Status 404 returned error can't find the container with id 2626ab2a2ac683864c0963f9846908a154239e59df5d45fbb2096fac1d5dcb07
Jul 22 01:52:53.071885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:53.071851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c" event={"ID":"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f","Type":"ContainerStarted","Data":"2626ab2a2ac683864c0963f9846908a154239e59df5d45fbb2096fac1d5dcb07"}
Jul 22 01:52:58.100110 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:58.100076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c" event={"ID":"2cd6b3e8-ed8b-4d34-9247-b07dad382d1f","Type":"ContainerStarted","Data":"435bbb83185df119d46e4d04b73337b0d033ee182ed310456e80e7b9bfa355da"}
Jul 22 01:52:58.100490 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:58.100204    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:52:58.116314 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:52:58.116270    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c" podStartSLOduration=1.1470964989999999 podStartE2EDuration="6.116256876s" totalImagesPullingTime="4.969160377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:52:58.114917131 +0000 UTC m=+489.617775361" watchObservedRunningTime="2026-07-22 01:52:58.116256876 +0000 UTC m=+489.619115104"
Jul 22 01:53:04.132721 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.132693    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-7j62c"
Jul 22 01:53:04.707363 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.707323    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-shsv5"]
Jul 22 01:53:04.714011 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.713986    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:04.716506 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.716483    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-24slz\""
Jul 22 01:53:04.716623 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.716562    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-shsv5"]
Jul 22 01:53:04.879432 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.879393    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-shsv5"]
Jul 22 01:53:04.879670 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:53:04.879650    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bnwk6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-shsv5" podUID="1230adfa-7edb-465e-85a1-685452a924e0"
Jul 22 01:53:04.892045 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.892020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwk6\" (UniqueName: \"kubernetes.io/projected/1230adfa-7edb-465e-85a1-685452a924e0-kube-api-access-bnwk6\") pod \"authorino-5d4b67c446-shsv5\" (UID: \"1230adfa-7edb-465e-85a1-685452a924e0\") " pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:04.901846 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.901823    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-9lt84"]
Jul 22 01:53:04.905835 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.905815    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:04.908169 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.908153    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 22 01:53:04.912436 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.912412    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-9lt84"]
Jul 22 01:53:04.957864 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.957787    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-9lt84"]
Jul 22 01:53:04.958053 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:53:04.958028    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2mrbm tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-9lt84" podUID="259efcc3-f31b-4e61-95e0-93fef092fd9b"
Jul 22 01:53:04.986339 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.986307    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5474977bf5-xlppt"]
Jul 22 01:53:04.990251 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.990225    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:04.992724 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.992698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwk6\" (UniqueName: \"kubernetes.io/projected/1230adfa-7edb-465e-85a1-685452a924e0-kube-api-access-bnwk6\") pod \"authorino-5d4b67c446-shsv5\" (UID: \"1230adfa-7edb-465e-85a1-685452a924e0\") " pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:04.992830 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.992774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/259efcc3-f31b-4e61-95e0-93fef092fd9b-tls-cert\") pod \"authorino-76877994-9lt84\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") " pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:04.992830 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.992817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrbm\" (UniqueName: \"kubernetes.io/projected/259efcc3-f31b-4e61-95e0-93fef092fd9b-kube-api-access-2mrbm\") pod \"authorino-76877994-9lt84\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") " pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:04.999234 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:04.999209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5474977bf5-xlppt"]
Jul 22 01:53:05.004476 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.004450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwk6\" (UniqueName: \"kubernetes.io/projected/1230adfa-7edb-465e-85a1-685452a924e0-kube-api-access-bnwk6\") pod \"authorino-5d4b67c446-shsv5\" (UID: \"1230adfa-7edb-465e-85a1-685452a924e0\") " pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:05.094130 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.094094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/259efcc3-f31b-4e61-95e0-93fef092fd9b-tls-cert\") pod \"authorino-76877994-9lt84\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") " pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:05.094130 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.094135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrbm\" (UniqueName: \"kubernetes.io/projected/259efcc3-f31b-4e61-95e0-93fef092fd9b-kube-api-access-2mrbm\") pod \"authorino-76877994-9lt84\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") " pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:05.094330 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.094155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2tcz\" (UniqueName: \"kubernetes.io/projected/84157b8a-1847-4561-a43f-73a24a59f9fb-kube-api-access-p2tcz\") pod \"authorino-5474977bf5-xlppt\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") " pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.094330 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.094191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84157b8a-1847-4561-a43f-73a24a59f9fb-tls-cert\") pod \"authorino-5474977bf5-xlppt\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") " pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.096457 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.096427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/259efcc3-f31b-4e61-95e0-93fef092fd9b-tls-cert\") pod \"authorino-76877994-9lt84\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") " pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:05.102425 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.102400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrbm\" (UniqueName: \"kubernetes.io/projected/259efcc3-f31b-4e61-95e0-93fef092fd9b-kube-api-access-2mrbm\") pod \"authorino-76877994-9lt84\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") " pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:05.126615 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.126593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:05.126751 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.126593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:05.134163 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.134146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:05.136718 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.136701    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:05.194591 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.194557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2tcz\" (UniqueName: \"kubernetes.io/projected/84157b8a-1847-4561-a43f-73a24a59f9fb-kube-api-access-p2tcz\") pod \"authorino-5474977bf5-xlppt\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") " pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.194802 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.194615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84157b8a-1847-4561-a43f-73a24a59f9fb-tls-cert\") pod \"authorino-5474977bf5-xlppt\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") " pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.197022 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.196999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84157b8a-1847-4561-a43f-73a24a59f9fb-tls-cert\") pod \"authorino-5474977bf5-xlppt\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") " pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.202297 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.202264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2tcz\" (UniqueName: \"kubernetes.io/projected/84157b8a-1847-4561-a43f-73a24a59f9fb-kube-api-access-p2tcz\") pod \"authorino-5474977bf5-xlppt\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") " pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.295258 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.295229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnwk6\" (UniqueName: \"kubernetes.io/projected/1230adfa-7edb-465e-85a1-685452a924e0-kube-api-access-bnwk6\") pod \"1230adfa-7edb-465e-85a1-685452a924e0\" (UID: \"1230adfa-7edb-465e-85a1-685452a924e0\") "
Jul 22 01:53:05.295416 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.295269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/259efcc3-f31b-4e61-95e0-93fef092fd9b-tls-cert\") pod \"259efcc3-f31b-4e61-95e0-93fef092fd9b\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") "
Jul 22 01:53:05.295416 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.295309    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mrbm\" (UniqueName: \"kubernetes.io/projected/259efcc3-f31b-4e61-95e0-93fef092fd9b-kube-api-access-2mrbm\") pod \"259efcc3-f31b-4e61-95e0-93fef092fd9b\" (UID: \"259efcc3-f31b-4e61-95e0-93fef092fd9b\") "
Jul 22 01:53:05.297430 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.297400    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259efcc3-f31b-4e61-95e0-93fef092fd9b-kube-api-access-2mrbm" (OuterVolumeSpecName: "kube-api-access-2mrbm") pod "259efcc3-f31b-4e61-95e0-93fef092fd9b" (UID: "259efcc3-f31b-4e61-95e0-93fef092fd9b"). InnerVolumeSpecName "kube-api-access-2mrbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:53:05.297430 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.297411    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1230adfa-7edb-465e-85a1-685452a924e0-kube-api-access-bnwk6" (OuterVolumeSpecName: "kube-api-access-bnwk6") pod "1230adfa-7edb-465e-85a1-685452a924e0" (UID: "1230adfa-7edb-465e-85a1-685452a924e0"). InnerVolumeSpecName "kube-api-access-bnwk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:53:05.297580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.297432    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259efcc3-f31b-4e61-95e0-93fef092fd9b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "259efcc3-f31b-4e61-95e0-93fef092fd9b" (UID: "259efcc3-f31b-4e61-95e0-93fef092fd9b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:53:05.300025 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.300010    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:53:05.395799 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.395774    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnwk6\" (UniqueName: \"kubernetes.io/projected/1230adfa-7edb-465e-85a1-685452a924e0-kube-api-access-bnwk6\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:53:05.395799 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.395798    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/259efcc3-f31b-4e61-95e0-93fef092fd9b-tls-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:53:05.395972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.395811    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrbm\" (UniqueName: \"kubernetes.io/projected/259efcc3-f31b-4e61-95e0-93fef092fd9b-kube-api-access-2mrbm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:53:05.424795 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:05.424770    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5474977bf5-xlppt"]
Jul 22 01:53:05.427008 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:53:05.426975    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84157b8a_1847_4561_a43f_73a24a59f9fb.slice/crio-eee4967e17e0c63a9f30a839710ab26188066cd31e37288afb6cd515b4ff58cf WatchSource:0}: Error finding container eee4967e17e0c63a9f30a839710ab26188066cd31e37288afb6cd515b4ff58cf: Status 404 returned error can't find the container with id eee4967e17e0c63a9f30a839710ab26188066cd31e37288afb6cd515b4ff58cf
Jul 22 01:53:06.131856 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.131761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5474977bf5-xlppt" event={"ID":"84157b8a-1847-4561-a43f-73a24a59f9fb","Type":"ContainerStarted","Data":"80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957"}
Jul 22 01:53:06.131856 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.131811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5474977bf5-xlppt" event={"ID":"84157b8a-1847-4561-a43f-73a24a59f9fb","Type":"ContainerStarted","Data":"eee4967e17e0c63a9f30a839710ab26188066cd31e37288afb6cd515b4ff58cf"}
Jul 22 01:53:06.131856 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.131821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-9lt84"
Jul 22 01:53:06.132139 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.131826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-shsv5"
Jul 22 01:53:06.148359 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.148318    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5474977bf5-xlppt" podStartSLOduration=1.748519791 podStartE2EDuration="2.148306015s" totalImagesPullingTime="399.786224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:06.146791389 +0000 UTC m=+497.649649615" watchObservedRunningTime="2026-07-22 01:53:06.148306015 +0000 UTC m=+497.651164243"
Jul 22 01:53:06.173179 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.173144    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-shsv5"]
Jul 22 01:53:06.181610 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.181583    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-shsv5"]
Jul 22 01:53:06.206214 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.206184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-9lt84"]
Jul 22 01:53:06.209535 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:06.209509    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-9lt84"]
Jul 22 01:53:07.102411 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:07.102377    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1230adfa-7edb-465e-85a1-685452a924e0" path="/var/lib/kubelet/pods/1230adfa-7edb-465e-85a1-685452a924e0/volumes"
Jul 22 01:53:07.102706 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:07.102690    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259efcc3-f31b-4e61-95e0-93fef092fd9b" path="/var/lib/kubelet/pods/259efcc3-f31b-4e61-95e0-93fef092fd9b/volumes"
Jul 22 01:53:11.348916 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.348872    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-969c84445-z7t6q"]
Jul 22 01:53:11.375741 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.375695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-969c84445-z7t6q"]
Jul 22 01:53:11.375886 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.375819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.378385 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.378365    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-f48b9\""
Jul 22 01:53:11.378524 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.378366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 22 01:53:11.379400 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.379380    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 22 01:53:11.445797 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.445762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cwf\" (UniqueName: \"kubernetes.io/projected/16ec83b1-6946-4772-a027-19c74a090a3c-kube-api-access-d8cwf\") pod \"maas-controller-969c84445-z7t6q\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") " pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.445955 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.445857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16ec83b1-6946-4772-a027-19c74a090a3c-webhook-cert\") pod \"maas-controller-969c84445-z7t6q\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") " pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.546795 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.546758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cwf\" (UniqueName: \"kubernetes.io/projected/16ec83b1-6946-4772-a027-19c74a090a3c-kube-api-access-d8cwf\") pod \"maas-controller-969c84445-z7t6q\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") " pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.546972 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.546839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16ec83b1-6946-4772-a027-19c74a090a3c-webhook-cert\") pod \"maas-controller-969c84445-z7t6q\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") " pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.549266 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.549238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16ec83b1-6946-4772-a027-19c74a090a3c-webhook-cert\") pod \"maas-controller-969c84445-z7t6q\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") " pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.555624 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.555592    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cwf\" (UniqueName: \"kubernetes.io/projected/16ec83b1-6946-4772-a027-19c74a090a3c-kube-api-access-d8cwf\") pod \"maas-controller-969c84445-z7t6q\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") " pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.686168 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.686087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:11.814232 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:11.814207    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-969c84445-z7t6q"]
Jul 22 01:53:11.816185 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:53:11.816162    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ec83b1_6946_4772_a027_19c74a090a3c.slice/crio-d98aa60dabad0fe9ca0ae4ca60fc2bc0721384597667c8692fd6387b06a600ad WatchSource:0}: Error finding container d98aa60dabad0fe9ca0ae4ca60fc2bc0721384597667c8692fd6387b06a600ad: Status 404 returned error can't find the container with id d98aa60dabad0fe9ca0ae4ca60fc2bc0721384597667c8692fd6387b06a600ad
Jul 22 01:53:12.156033 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:12.155997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-z7t6q" event={"ID":"16ec83b1-6946-4772-a027-19c74a090a3c","Type":"ContainerStarted","Data":"d98aa60dabad0fe9ca0ae4ca60fc2bc0721384597667c8692fd6387b06a600ad"}
Jul 22 01:53:14.166521 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:14.166488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-z7t6q" event={"ID":"16ec83b1-6946-4772-a027-19c74a090a3c","Type":"ContainerStarted","Data":"39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102"}
Jul 22 01:53:14.166902 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:14.166591    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:14.185682 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:14.185623    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-969c84445-z7t6q" podStartSLOduration=0.93586448 podStartE2EDuration="3.185608911s" totalImagesPullingTime="2.249744431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:14.184321461 +0000 UTC m=+505.687179719" watchObservedRunningTime="2026-07-22 01:53:14.185608911 +0000 UTC m=+505.688467169"
Jul 22 01:53:25.173386 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:25.173357    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:53:47.424581 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.424485    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"]
Jul 22 01:53:47.432164 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.432140    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.434612 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.434589    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 22 01:53:47.434848 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.434831    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hhprp\""
Jul 22 01:53:47.439415 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.439390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"]
Jul 22 01:53:47.533230 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.533197    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef78394a-1093-4b11-9be4-2197125d314c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-cmc7n\" (UID: \"ef78394a-1093-4b11-9be4-2197125d314c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.533400 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.533271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78mg\" (UniqueName: \"kubernetes.io/projected/ef78394a-1093-4b11-9be4-2197125d314c-kube-api-access-f78mg\") pod \"payload-pre-processing-7677b774bb-cmc7n\" (UID: \"ef78394a-1093-4b11-9be4-2197125d314c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.634562 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.634525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78mg\" (UniqueName: \"kubernetes.io/projected/ef78394a-1093-4b11-9be4-2197125d314c-kube-api-access-f78mg\") pod \"payload-pre-processing-7677b774bb-cmc7n\" (UID: \"ef78394a-1093-4b11-9be4-2197125d314c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.634751 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.634570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef78394a-1093-4b11-9be4-2197125d314c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-cmc7n\" (UID: \"ef78394a-1093-4b11-9be4-2197125d314c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.635263 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.635245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef78394a-1093-4b11-9be4-2197125d314c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-cmc7n\" (UID: \"ef78394a-1093-4b11-9be4-2197125d314c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.642667 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.642637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78mg\" (UniqueName: \"kubernetes.io/projected/ef78394a-1093-4b11-9be4-2197125d314c-kube-api-access-f78mg\") pod \"payload-pre-processing-7677b774bb-cmc7n\" (UID: \"ef78394a-1093-4b11-9be4-2197125d314c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.744893 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.744794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:47.873821 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:47.873792    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"]
Jul 22 01:53:47.876422 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:53:47.876397    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef78394a_1093_4b11_9be4_2197125d314c.slice/crio-9dba54ea9c1fe2bd1cd6cb8e0d1f3238203256b90918367298e698e831bbcac7 WatchSource:0}: Error finding container 9dba54ea9c1fe2bd1cd6cb8e0d1f3238203256b90918367298e698e831bbcac7: Status 404 returned error can't find the container with id 9dba54ea9c1fe2bd1cd6cb8e0d1f3238203256b90918367298e698e831bbcac7
Jul 22 01:53:48.306415 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:48.306373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n" event={"ID":"ef78394a-1093-4b11-9be4-2197125d314c","Type":"ContainerStarted","Data":"9dba54ea9c1fe2bd1cd6cb8e0d1f3238203256b90918367298e698e831bbcac7"}
Jul 22 01:53:50.316386 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:50.316294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n" event={"ID":"ef78394a-1093-4b11-9be4-2197125d314c","Type":"ContainerStarted","Data":"76a540b4e64516178594a0db064a959ac83bd795234e6a43496a3a658e5ee58b"}
Jul 22 01:53:50.316766 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:50.316505    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:53:50.331658 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:53:50.331616    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n" podStartSLOduration=1.155592261 podStartE2EDuration="3.331603157s" totalImagesPullingTime="2.176010896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:53:50.33131299 +0000 UTC m=+541.834171217" watchObservedRunningTime="2026-07-22 01:53:50.331603157 +0000 UTC m=+541.834461386"
Jul 22 01:54:01.323464 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:01.323435    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-cmc7n"
Jul 22 01:54:03.109132 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.109098    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"]
Jul 22 01:54:03.113269 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.113250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.116094 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.116072    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 22 01:54:03.116197 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.116070    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 22 01:54:03.116979 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.116965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 22 01:54:03.117059 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.116970    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8bf95\""
Jul 22 01:54:03.121590 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.121569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"]
Jul 22 01:54:03.268320 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.268271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.268529 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.268347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.268604 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.268518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7mwl\" (UniqueName: \"kubernetes.io/projected/0ace246d-5612-42b5-af14-37c2f9af6329-kube-api-access-s7mwl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.268771 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.268717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.268896 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.268855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ace246d-5612-42b5-af14-37c2f9af6329-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.268896 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.268885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370196 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ace246d-5612-42b5-af14-37c2f9af6329-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370196 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370196 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7mwl\" (UniqueName: \"kubernetes.io/projected/0ace246d-5612-42b5-af14-37c2f9af6329-kube-api-access-s7mwl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370484 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370645 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370645 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.370715 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.370698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.372519 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.372495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ace246d-5612-42b5-af14-37c2f9af6329-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.372808 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.372793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ace246d-5612-42b5-af14-37c2f9af6329-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.378761 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.378702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7mwl\" (UniqueName: \"kubernetes.io/projected/0ace246d-5612-42b5-af14-37c2f9af6329-kube-api-access-s7mwl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j\" (UID: \"0ace246d-5612-42b5-af14-37c2f9af6329\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.424917 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.424875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:03.556354 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:03.556290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"]
Jul 22 01:54:03.559130 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:54:03.559105    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ace246d_5612_42b5_af14_37c2f9af6329.slice/crio-094e7a9e979a2dc81a3693d9af3d2938dd3947a2c385ce71fece05c1eee1f0ad WatchSource:0}: Error finding container 094e7a9e979a2dc81a3693d9af3d2938dd3947a2c385ce71fece05c1eee1f0ad: Status 404 returned error can't find the container with id 094e7a9e979a2dc81a3693d9af3d2938dd3947a2c385ce71fece05c1eee1f0ad
Jul 22 01:54:04.387641 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:04.387603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j" event={"ID":"0ace246d-5612-42b5-af14-37c2f9af6329","Type":"ContainerStarted","Data":"094e7a9e979a2dc81a3693d9af3d2938dd3947a2c385ce71fece05c1eee1f0ad"}
Jul 22 01:54:05.396693 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:05.396661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j" event={"ID":"0ace246d-5612-42b5-af14-37c2f9af6329","Type":"ContainerStarted","Data":"564c5dfcb68df5c269ad116d5705c72d3f4e331ee256bb3accf1d57c0d6dbb51"}
Jul 22 01:54:05.414476 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:05.414425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j" podStartSLOduration=1.3084635709999999 podStartE2EDuration="2.414410069s" totalImagesPullingTime="1.105946498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:05.411916147 +0000 UTC m=+556.914774372" watchObservedRunningTime="2026-07-22 01:54:05.414410069 +0000 UTC m=+556.917268298"
Jul 22 01:54:13.425322 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:13.425282    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:13.425779 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:13.425352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:13.438197 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:13.438175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:14.449124 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:14.449093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j"
Jul 22 01:54:35.476282 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.476249    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"]
Jul 22 01:54:35.484932 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.484905    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.487601 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.487406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 22 01:54:35.491412 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.491389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"]
Jul 22 01:54:35.536514 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.536480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.536675 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.536543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.536675 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.536569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438bea22-072e-49b6-9957-97b4bd5f83c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.536799 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.536673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.536799 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.536715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c8k4\" (UniqueName: \"kubernetes.io/projected/438bea22-072e-49b6-9957-97b4bd5f83c6-kube-api-access-2c8k4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.536873 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.536808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637355 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c8k4\" (UniqueName: \"kubernetes.io/projected/438bea22-072e-49b6-9957-97b4bd5f83c6-kube-api-access-2c8k4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637533 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637666 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637746 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637746 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637699    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438bea22-072e-49b6-9957-97b4bd5f83c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637858 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637914 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.637973 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.637931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.639927 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.639896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/438bea22-072e-49b6-9957-97b4bd5f83c6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.640438 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.640409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/438bea22-072e-49b6-9957-97b4bd5f83c6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.644984 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.644966    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c8k4\" (UniqueName: \"kubernetes.io/projected/438bea22-072e-49b6-9957-97b4bd5f83c6-kube-api-access-2c8k4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kv6gc\" (UID: \"438bea22-072e-49b6-9957-97b4bd5f83c6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.797680 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.797642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:35.929581 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:35.929551    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"]
Jul 22 01:54:35.931500 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:54:35.931459    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438bea22_072e_49b6_9957_97b4bd5f83c6.slice/crio-ce324e975a23459b7cbc4a4e95263bcbf7e9fd6cf16e2d8ec0cee0fdf65eddba WatchSource:0}: Error finding container ce324e975a23459b7cbc4a4e95263bcbf7e9fd6cf16e2d8ec0cee0fdf65eddba: Status 404 returned error can't find the container with id ce324e975a23459b7cbc4a4e95263bcbf7e9fd6cf16e2d8ec0cee0fdf65eddba
Jul 22 01:54:36.542374 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:36.542340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc" event={"ID":"438bea22-072e-49b6-9957-97b4bd5f83c6","Type":"ContainerStarted","Data":"51d394cc5bb80646fec1f7dc8001ed41f63551832775e2ac6ecee6afb236b3e3"}
Jul 22 01:54:36.542766 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:36.542399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc" event={"ID":"438bea22-072e-49b6-9957-97b4bd5f83c6","Type":"ContainerStarted","Data":"ce324e975a23459b7cbc4a4e95263bcbf7e9fd6cf16e2d8ec0cee0fdf65eddba"}
Jul 22 01:54:36.562095 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:36.562051    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc" podStartSLOduration=1.310898783 podStartE2EDuration="1.562035537s" totalImagesPullingTime="251.136754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:36.560572906 +0000 UTC m=+588.063431135" watchObservedRunningTime="2026-07-22 01:54:36.562035537 +0000 UTC m=+588.064893766"
Jul 22 01:54:42.385577 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.385548    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"]
Jul 22 01:54:42.389377 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.389356    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.392020 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.392000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 22 01:54:42.399026 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.399000    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"]
Jul 22 01:54:42.498312 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.498280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.498312 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.498319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.498493 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.498411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.498493 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.498439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624f147e-05a1-459e-99ee-47d45e56c1ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.498564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.498514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.498564 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.498531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22vt\" (UniqueName: \"kubernetes.io/projected/624f147e-05a1-459e-99ee-47d45e56c1ff-kube-api-access-z22vt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599140 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599140 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22vt\" (UniqueName: \"kubernetes.io/projected/624f147e-05a1-459e-99ee-47d45e56c1ff-kube-api-access-z22vt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599377 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599377 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599182    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599377 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599214    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599377 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624f147e-05a1-459e-99ee-47d45e56c1ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599582 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599641 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599612    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.599679 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.599642    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.601501 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.601480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/624f147e-05a1-459e-99ee-47d45e56c1ff-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.601759 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.601743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/624f147e-05a1-459e-99ee-47d45e56c1ff-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.607233 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.607211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22vt\" (UniqueName: \"kubernetes.io/projected/624f147e-05a1-459e-99ee-47d45e56c1ff-kube-api-access-z22vt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz\" (UID: \"624f147e-05a1-459e-99ee-47d45e56c1ff\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.699598 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.699528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:42.831123 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:42.831096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"]
Jul 22 01:54:42.834004 ip-10-0-130-189 kubenswrapper[2561]: W0722 01:54:42.833974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod624f147e_05a1_459e_99ee_47d45e56c1ff.slice/crio-da20586c8cc633ef12d1546381b658098bcb96818feeb7f125a4645ad5878c26 WatchSource:0}: Error finding container da20586c8cc633ef12d1546381b658098bcb96818feeb7f125a4645ad5878c26: Status 404 returned error can't find the container with id da20586c8cc633ef12d1546381b658098bcb96818feeb7f125a4645ad5878c26
Jul 22 01:54:43.575480 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:43.575448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz" event={"ID":"624f147e-05a1-459e-99ee-47d45e56c1ff","Type":"ContainerStarted","Data":"d210ec873bbb561b6b19f236b3c74e238495a197bd47c1bf34f81d8d696f70fb"}
Jul 22 01:54:43.575480 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:43.575484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz" event={"ID":"624f147e-05a1-459e-99ee-47d45e56c1ff","Type":"ContainerStarted","Data":"da20586c8cc633ef12d1546381b658098bcb96818feeb7f125a4645ad5878c26"}
Jul 22 01:54:43.592505 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:43.592459    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz" podStartSLOduration=1.408082918 podStartE2EDuration="1.592445908s" totalImagesPullingTime="184.36299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 01:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 01:54:43.591086486 +0000 UTC m=+595.093944715" watchObservedRunningTime="2026-07-22 01:54:43.592445908 +0000 UTC m=+595.095304137"
Jul 22 01:54:45.798580 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:45.798545    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:45.799002 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:45.798602    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:45.811049 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:45.811026    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:46.602997 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:46.602970    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kv6gc"
Jul 22 01:54:48.970511 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:48.970485    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:54:48.971054 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:48.971034    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:54:52.700056 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:52.700022    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:52.702885 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:52.700064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:52.712514 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:52.712493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:54:53.635061 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:54:53.635033    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz"
Jul 22 01:55:49.361041 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.361008    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5474977bf5-xlppt"]
Jul 22 01:55:49.361470 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.361207    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5474977bf5-xlppt" podUID="84157b8a-1847-4561-a43f-73a24a59f9fb" containerName="authorino" containerID="cri-o://80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957" gracePeriod=30
Jul 22 01:55:49.605081 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.605060    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:55:49.778435 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.778407    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2tcz\" (UniqueName: \"kubernetes.io/projected/84157b8a-1847-4561-a43f-73a24a59f9fb-kube-api-access-p2tcz\") pod \"84157b8a-1847-4561-a43f-73a24a59f9fb\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") "
Jul 22 01:55:49.778583 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.778442    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84157b8a-1847-4561-a43f-73a24a59f9fb-tls-cert\") pod \"84157b8a-1847-4561-a43f-73a24a59f9fb\" (UID: \"84157b8a-1847-4561-a43f-73a24a59f9fb\") "
Jul 22 01:55:49.780426 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.780397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84157b8a-1847-4561-a43f-73a24a59f9fb-kube-api-access-p2tcz" (OuterVolumeSpecName: "kube-api-access-p2tcz") pod "84157b8a-1847-4561-a43f-73a24a59f9fb" (UID: "84157b8a-1847-4561-a43f-73a24a59f9fb"). InnerVolumeSpecName "kube-api-access-p2tcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:55:49.787876 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.787855    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84157b8a-1847-4561-a43f-73a24a59f9fb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "84157b8a-1847-4561-a43f-73a24a59f9fb" (UID: "84157b8a-1847-4561-a43f-73a24a59f9fb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:55:49.877189 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.877155    2561 generic.go:356] "Generic (PLEG): container finished" podID="84157b8a-1847-4561-a43f-73a24a59f9fb" containerID="80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957" exitCode=0
Jul 22 01:55:49.877340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.877197    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5474977bf5-xlppt"
Jul 22 01:55:49.877340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.877212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5474977bf5-xlppt" event={"ID":"84157b8a-1847-4561-a43f-73a24a59f9fb","Type":"ContainerDied","Data":"80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957"}
Jul 22 01:55:49.877340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.877249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5474977bf5-xlppt" event={"ID":"84157b8a-1847-4561-a43f-73a24a59f9fb","Type":"ContainerDied","Data":"eee4967e17e0c63a9f30a839710ab26188066cd31e37288afb6cd515b4ff58cf"}
Jul 22 01:55:49.877340 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.877273    2561 scope.go:117] "RemoveContainer" containerID="80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957"
Jul 22 01:55:49.878975 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.878951    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2tcz\" (UniqueName: \"kubernetes.io/projected/84157b8a-1847-4561-a43f-73a24a59f9fb-kube-api-access-p2tcz\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:55:49.878975 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.878976    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84157b8a-1847-4561-a43f-73a24a59f9fb-tls-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:55:49.885964 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.885947    2561 scope.go:117] "RemoveContainer" containerID="80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957"
Jul 22 01:55:49.886215 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:55:49.886197    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957\": container with ID starting with 80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957 not found: ID does not exist" containerID="80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957"
Jul 22 01:55:49.886262 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.886225    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957"} err="failed to get container status \"80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957\": rpc error: code = NotFound desc = could not find container \"80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957\": container with ID starting with 80b5978eff2ebbda70f0f916ad2a54f8a510d48777582d1ac793dc063d18c957 not found: ID does not exist"
Jul 22 01:55:49.898055 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.898019    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5474977bf5-xlppt"]
Jul 22 01:55:49.901515 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:49.901494    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5474977bf5-xlppt"]
Jul 22 01:55:51.102325 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:55:51.102291    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84157b8a-1847-4561-a43f-73a24a59f9fb" path="/var/lib/kubelet/pods/84157b8a-1847-4561-a43f-73a24a59f9fb/volumes"
Jul 22 01:56:36.773250 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:36.773216    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-969c84445-z7t6q"]
Jul 22 01:56:36.773763 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:36.773412    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-969c84445-z7t6q" podUID="16ec83b1-6946-4772-a027-19c74a090a3c" containerName="manager" containerID="cri-o://39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102" gracePeriod=30
Jul 22 01:56:37.023658 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.023603    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:56:37.088622 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.088586    2561 generic.go:356] "Generic (PLEG): container finished" podID="16ec83b1-6946-4772-a027-19c74a090a3c" containerID="39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102" exitCode=0
Jul 22 01:56:37.088803 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.088628    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-969c84445-z7t6q"
Jul 22 01:56:37.088803 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.088673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-z7t6q" event={"ID":"16ec83b1-6946-4772-a027-19c74a090a3c","Type":"ContainerDied","Data":"39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102"}
Jul 22 01:56:37.088803 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.088722    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-969c84445-z7t6q" event={"ID":"16ec83b1-6946-4772-a027-19c74a090a3c","Type":"ContainerDied","Data":"d98aa60dabad0fe9ca0ae4ca60fc2bc0721384597667c8692fd6387b06a600ad"}
Jul 22 01:56:37.088803 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.088755    2561 scope.go:117] "RemoveContainer" containerID="39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102"
Jul 22 01:56:37.096825 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.096802    2561 scope.go:117] "RemoveContainer" containerID="39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102"
Jul 22 01:56:37.097081 ip-10-0-130-189 kubenswrapper[2561]: E0722 01:56:37.097058    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102\": container with ID starting with 39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102 not found: ID does not exist" containerID="39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102"
Jul 22 01:56:37.097177 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.097087    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102"} err="failed to get container status \"39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102\": rpc error: code = NotFound desc = could not find container \"39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102\": container with ID starting with 39103d3958010fa42c7a77aa5368c15d9e14b745eb978b479e5e055311328102 not found: ID does not exist"
Jul 22 01:56:37.160697 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.160655    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16ec83b1-6946-4772-a027-19c74a090a3c-webhook-cert\") pod \"16ec83b1-6946-4772-a027-19c74a090a3c\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") "
Jul 22 01:56:37.160883 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.160822    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8cwf\" (UniqueName: \"kubernetes.io/projected/16ec83b1-6946-4772-a027-19c74a090a3c-kube-api-access-d8cwf\") pod \"16ec83b1-6946-4772-a027-19c74a090a3c\" (UID: \"16ec83b1-6946-4772-a027-19c74a090a3c\") "
Jul 22 01:56:37.163023 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.162984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ec83b1-6946-4772-a027-19c74a090a3c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "16ec83b1-6946-4772-a027-19c74a090a3c" (UID: "16ec83b1-6946-4772-a027-19c74a090a3c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 01:56:37.163023 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.163005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ec83b1-6946-4772-a027-19c74a090a3c-kube-api-access-d8cwf" (OuterVolumeSpecName: "kube-api-access-d8cwf") pod "16ec83b1-6946-4772-a027-19c74a090a3c" (UID: "16ec83b1-6946-4772-a027-19c74a090a3c"). InnerVolumeSpecName "kube-api-access-d8cwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 01:56:37.262438 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.262408    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/16ec83b1-6946-4772-a027-19c74a090a3c-webhook-cert\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:56:37.262438 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.262434    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8cwf\" (UniqueName: \"kubernetes.io/projected/16ec83b1-6946-4772-a027-19c74a090a3c-kube-api-access-d8cwf\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 01:56:37.410664 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.410639    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-969c84445-z7t6q"]
Jul 22 01:56:37.416139 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:37.416117    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-969c84445-z7t6q"]
Jul 22 01:56:39.101584 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:56:39.101552    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ec83b1-6946-4772-a027-19c74a090a3c" path="/var/lib/kubelet/pods/16ec83b1-6946-4772-a027-19c74a090a3c/volumes"
Jul 22 01:59:48.995898 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:59:48.995876    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 01:59:48.996369 ip-10-0-130-189 kubenswrapper[2561]: I0722 01:59:48.996226    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:02:42.305918 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:42.305838    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"]
Jul 22 02:02:42.687597 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:42.687480    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4" podUID="aa5abf00-0010-40d6-be44-dbd3d3d24f8c" containerName="manager" containerID="cri-o://523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38" gracePeriod=10
Jul 22 02:02:45.146784 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.146757    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 02:02:45.240425 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.240341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccq6s\" (UniqueName: \"kubernetes.io/projected/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-kube-api-access-ccq6s\") pod \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") "
Jul 22 02:02:45.240565 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.240448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-extensions-socket-volume\") pod \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\" (UID: \"aa5abf00-0010-40d6-be44-dbd3d3d24f8c\") "
Jul 22 02:02:45.240939 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.240913    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aa5abf00-0010-40d6-be44-dbd3d3d24f8c" (UID: "aa5abf00-0010-40d6-be44-dbd3d3d24f8c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:02:45.242428 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.242402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-kube-api-access-ccq6s" (OuterVolumeSpecName: "kube-api-access-ccq6s") pod "aa5abf00-0010-40d6-be44-dbd3d3d24f8c" (UID: "aa5abf00-0010-40d6-be44-dbd3d3d24f8c"). InnerVolumeSpecName "kube-api-access-ccq6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:02:45.341696 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.341664    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccq6s\" (UniqueName: \"kubernetes.io/projected/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-kube-api-access-ccq6s\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:02:45.341696 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.341691    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa5abf00-0010-40d6-be44-dbd3d3d24f8c-extensions-socket-volume\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:02:45.701270 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.701233    2561 generic.go:356] "Generic (PLEG): container finished" podID="aa5abf00-0010-40d6-be44-dbd3d3d24f8c" containerID="523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38" exitCode=0
Jul 22 02:02:45.701438 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.701271    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"
Jul 22 02:02:45.701438 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.701296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4" event={"ID":"aa5abf00-0010-40d6-be44-dbd3d3d24f8c","Type":"ContainerDied","Data":"523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38"}
Jul 22 02:02:45.701438 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.701342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4" event={"ID":"aa5abf00-0010-40d6-be44-dbd3d3d24f8c","Type":"ContainerDied","Data":"4803baa4a990ce1c3d1eb2a9740799431770a8323c2b0deb43193530ed0abd40"}
Jul 22 02:02:45.701438 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.701368    2561 scope.go:117] "RemoveContainer" containerID="523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38"
Jul 22 02:02:45.710673 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.710653    2561 scope.go:117] "RemoveContainer" containerID="523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38"
Jul 22 02:02:45.710952 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:02:45.710931    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38\": container with ID starting with 523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38 not found: ID does not exist" containerID="523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38"
Jul 22 02:02:45.711007 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.710961    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38"} err="failed to get container status \"523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38\": rpc error: code = NotFound desc = could not find container \"523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38\": container with ID starting with 523241b2161b0d68cde64974cdb7709095d448161220fb376edbf92888cadb38 not found: ID does not exist"
Jul 22 02:02:45.723022 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.722999    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"]
Jul 22 02:02:45.725860 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:45.725841    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5g7b4"]
Jul 22 02:02:47.106907 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:02:47.106877    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5abf00-0010-40d6-be44-dbd3d3d24f8c" path="/var/lib/kubelet/pods/aa5abf00-0010-40d6-be44-dbd3d3d24f8c/volumes"
Jul 22 02:03:43.512722 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.512687    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"]
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513010    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84157b8a-1847-4561-a43f-73a24a59f9fb" containerName="authorino"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513021    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="84157b8a-1847-4561-a43f-73a24a59f9fb" containerName="authorino"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513033    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ec83b1-6946-4772-a027-19c74a090a3c" containerName="manager"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513038    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ec83b1-6946-4772-a027-19c74a090a3c" containerName="manager"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513047    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5abf00-0010-40d6-be44-dbd3d3d24f8c" containerName="manager"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513053    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5abf00-0010-40d6-be44-dbd3d3d24f8c" containerName="manager"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513106    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="84157b8a-1847-4561-a43f-73a24a59f9fb" containerName="authorino"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513113    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5abf00-0010-40d6-be44-dbd3d3d24f8c" containerName="manager"
Jul 22 02:03:43.513227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.513121    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ec83b1-6946-4772-a027-19c74a090a3c" containerName="manager"
Jul 22 02:03:43.515963 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.515945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.518511 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.518483    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l7cfb\""
Jul 22 02:03:43.527106 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.527084    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"]
Jul 22 02:03:43.620013 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.619986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4dn\" (UniqueName: \"kubernetes.io/projected/5fce6dbc-c000-4506-a27a-bbcd4d32cec3-kube-api-access-jk4dn\") pod \"kuadrant-operator-controller-manager-86878c769f-rs68l\" (UID: \"5fce6dbc-c000-4506-a27a-bbcd4d32cec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.620111 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.620020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fce6dbc-c000-4506-a27a-bbcd4d32cec3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rs68l\" (UID: \"5fce6dbc-c000-4506-a27a-bbcd4d32cec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.720913 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.720880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4dn\" (UniqueName: \"kubernetes.io/projected/5fce6dbc-c000-4506-a27a-bbcd4d32cec3-kube-api-access-jk4dn\") pod \"kuadrant-operator-controller-manager-86878c769f-rs68l\" (UID: \"5fce6dbc-c000-4506-a27a-bbcd4d32cec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.721026 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.720923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fce6dbc-c000-4506-a27a-bbcd4d32cec3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rs68l\" (UID: \"5fce6dbc-c000-4506-a27a-bbcd4d32cec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.721231 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.721217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fce6dbc-c000-4506-a27a-bbcd4d32cec3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rs68l\" (UID: \"5fce6dbc-c000-4506-a27a-bbcd4d32cec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.729712 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.729685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4dn\" (UniqueName: \"kubernetes.io/projected/5fce6dbc-c000-4506-a27a-bbcd4d32cec3-kube-api-access-jk4dn\") pod \"kuadrant-operator-controller-manager-86878c769f-rs68l\" (UID: \"5fce6dbc-c000-4506-a27a-bbcd4d32cec3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.825555 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.825459    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:43.972630 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:43.972607    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"]
Jul 22 02:03:43.975075 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:03:43.975037    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fce6dbc_c000_4506_a27a_bbcd4d32cec3.slice/crio-30a07c03665e78a2a17f1b46731d7017b232d7f8ef9e52db312946dfd382cd3e WatchSource:0}: Error finding container 30a07c03665e78a2a17f1b46731d7017b232d7f8ef9e52db312946dfd382cd3e: Status 404 returned error can't find the container with id 30a07c03665e78a2a17f1b46731d7017b232d7f8ef9e52db312946dfd382cd3e
Jul 22 02:03:44.955000 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:44.954967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l" event={"ID":"5fce6dbc-c000-4506-a27a-bbcd4d32cec3","Type":"ContainerStarted","Data":"4de14e6f912e8140ea2f0272589e969496f9a9615100f586864296b45aced1bc"}
Jul 22 02:03:44.955377 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:44.955019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l" event={"ID":"5fce6dbc-c000-4506-a27a-bbcd4d32cec3","Type":"ContainerStarted","Data":"30a07c03665e78a2a17f1b46731d7017b232d7f8ef9e52db312946dfd382cd3e"}
Jul 22 02:03:44.955377 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:44.955049    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:03:44.974672 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:44.974628    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l" podStartSLOduration=1.974609918 podStartE2EDuration="1.974609918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:03:44.972096474 +0000 UTC m=+1136.474954718" watchObservedRunningTime="2026-07-22 02:03:44.974609918 +0000 UTC m=+1136.477468148"
Jul 22 02:03:55.962286 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:03:55.962258    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rs68l"
Jul 22 02:04:49.020382 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:04:49.020358    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:04:49.021409 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:04:49.021391    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:06:25.404125 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.404088    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"]
Jul 22 02:06:25.407898 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.407879    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.410378 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.410354    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c838639d-kserve-self-signed-certs\""
Jul 22 02:06:25.421086 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.421061    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"]
Jul 22 02:06:25.476899 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.476860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-home\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.477050 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.476905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-model-cache\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.477050 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.476938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-dshm\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.477050 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.476967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8d551c-fe4b-441e-9013-df25a120033c-tls-certs\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.477050 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.477001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-tmp-dir\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.477050 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.477039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2f4s\" (UniqueName: \"kubernetes.io/projected/2c8d551c-fe4b-441e-9013-df25a120033c-kube-api-access-m2f4s\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578180 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-home\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578180 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-model-cache\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578431 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-dshm\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578431 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578223    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8d551c-fe4b-441e-9013-df25a120033c-tls-certs\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578431 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-tmp-dir\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578431 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2f4s\" (UniqueName: \"kubernetes.io/projected/2c8d551c-fe4b-441e-9013-df25a120033c-kube-api-access-m2f4s\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578684 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-home\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578684 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578666    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-model-cache\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.578872 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.578699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-tmp-dir\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.581009 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.580992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-dshm\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.581372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.581352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8d551c-fe4b-441e-9013-df25a120033c-tls-certs\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.586395 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.586374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2f4s\" (UniqueName: \"kubernetes.io/projected/2c8d551c-fe4b-441e-9013-df25a120033c-kube-api-access-m2f4s\") pod \"e2e-same-id-a-c838639d-kserve-f49787479-5kzfd\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") " pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.716606 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.716521    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:25.852166 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:25.852142    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"]
Jul 22 02:06:25.854059 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:06:25.854031    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c8d551c_fe4b_441e_9013_df25a120033c.slice/crio-1c1d22b1ca31a876205b9dfd6b0a4eeda599b77d333a8989ac84728acdc9a124 WatchSource:0}: Error finding container 1c1d22b1ca31a876205b9dfd6b0a4eeda599b77d333a8989ac84728acdc9a124: Status 404 returned error can't find the container with id 1c1d22b1ca31a876205b9dfd6b0a4eeda599b77d333a8989ac84728acdc9a124
Jul 22 02:06:26.671815 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:26.671775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd" event={"ID":"2c8d551c-fe4b-441e-9013-df25a120033c","Type":"ContainerStarted","Data":"c2d213f9dff697106c3565855185cd535e5ae125eac5329b1802675e498d118c"}
Jul 22 02:06:26.671815 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:26.671818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd" event={"ID":"2c8d551c-fe4b-441e-9013-df25a120033c","Type":"ContainerStarted","Data":"1c1d22b1ca31a876205b9dfd6b0a4eeda599b77d333a8989ac84728acdc9a124"}
Jul 22 02:06:26.691752 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:26.689532    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd" podStartSLOduration=1.689514415 podStartE2EDuration="1.689514415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:06:26.687167168 +0000 UTC m=+1298.190025399" watchObservedRunningTime="2026-07-22 02:06:26.689514415 +0000 UTC m=+1298.192372645"
Jul 22 02:06:31.492252 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.492219    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"]
Jul 22 02:06:31.495818 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.495795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.498282 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.498256    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c838639d-kserve-self-signed-certs\""
Jul 22 02:06:31.504915 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.504889    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"]
Jul 22 02:06:31.629989 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.629954    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25df628-ed7b-48cf-8243-42956c07902c-tls-certs\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.630149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.630025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-dshm\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.630149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.630067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-home\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.630149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.630094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-tmp-dir\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.630149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.630118    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkw2p\" (UniqueName: \"kubernetes.io/projected/d25df628-ed7b-48cf-8243-42956c07902c-kube-api-access-gkw2p\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.630149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.630141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-model-cache\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.730854 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.730821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkw2p\" (UniqueName: \"kubernetes.io/projected/d25df628-ed7b-48cf-8243-42956c07902c-kube-api-access-gkw2p\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.730861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-model-cache\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.730887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25df628-ed7b-48cf-8243-42956c07902c-tls-certs\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.730930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-dshm\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.730955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-home\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.730983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-tmp-dir\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731312 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.731288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-model-cache\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731385 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.731343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-tmp-dir\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.731444 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.731397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-home\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.733194 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.733172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-dshm\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.733584 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.733564    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25df628-ed7b-48cf-8243-42956c07902c-tls-certs\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.738575 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.738551    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkw2p\" (UniqueName: \"kubernetes.io/projected/d25df628-ed7b-48cf-8243-42956c07902c-kube-api-access-gkw2p\") pod \"e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") " pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.806382 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.806348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:31.939596 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:31.939565    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"]
Jul 22 02:06:31.941955 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:06:31.941928    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd25df628_ed7b_48cf_8243_42956c07902c.slice/crio-fbc3deebe336b5a316fe56dfabbb95bb7d566375b99de64bfeac4cb12f475ed6 WatchSource:0}: Error finding container fbc3deebe336b5a316fe56dfabbb95bb7d566375b99de64bfeac4cb12f475ed6: Status 404 returned error can't find the container with id fbc3deebe336b5a316fe56dfabbb95bb7d566375b99de64bfeac4cb12f475ed6
Jul 22 02:06:32.699451 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:32.699417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr" event={"ID":"d25df628-ed7b-48cf-8243-42956c07902c","Type":"ContainerStarted","Data":"5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43"}
Jul 22 02:06:32.699451 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:32.699457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr" event={"ID":"d25df628-ed7b-48cf-8243-42956c07902c","Type":"ContainerStarted","Data":"fbc3deebe336b5a316fe56dfabbb95bb7d566375b99de64bfeac4cb12f475ed6"}
Jul 22 02:06:32.722512 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:32.719307    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr" podStartSLOduration=1.719287856 podStartE2EDuration="1.719287856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:06:32.716328034 +0000 UTC m=+1304.219186263" watchObservedRunningTime="2026-07-22 02:06:32.719287856 +0000 UTC m=+1304.222146085"
Jul 22 02:06:35.717398 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:35.717317    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:35.717398 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:35.717363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:35.729511 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:35.729492    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:36.727858 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:36.727822    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:06:41.807070 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:41.807034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:41.807482 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:41.807175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:41.819302 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:41.819275    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:42.756227 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:42.756201    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:06:55.693821 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:55.693789    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"]
Jul 22 02:06:57.015312 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:06:57.015281    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"]
Jul 22 02:07:10.714246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:10.714178    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd" podUID="2c8d551c-fe4b-441e-9013-df25a120033c" containerName="main" containerID="cri-o://c2d213f9dff697106c3565855185cd535e5ae125eac5329b1802675e498d118c" gracePeriod=45
Jul 22 02:07:10.874170 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:10.874139    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c8d551c-fe4b-441e-9013-df25a120033c" containerID="c2d213f9dff697106c3565855185cd535e5ae125eac5329b1802675e498d118c" exitCode=0
Jul 22 02:07:10.874325 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:10.874190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd" event={"ID":"2c8d551c-fe4b-441e-9013-df25a120033c","Type":"ContainerDied","Data":"c2d213f9dff697106c3565855185cd535e5ae125eac5329b1802675e498d118c"}
Jul 22 02:07:10.958988 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:10.958962    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:07:11.060600 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-home\") pod \"2c8d551c-fe4b-441e-9013-df25a120033c\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") "
Jul 22 02:07:11.060600 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060598    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-tmp-dir\") pod \"2c8d551c-fe4b-441e-9013-df25a120033c\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") "
Jul 22 02:07:11.060872 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-model-cache\") pod \"2c8d551c-fe4b-441e-9013-df25a120033c\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") "
Jul 22 02:07:11.060872 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060666    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-dshm\") pod \"2c8d551c-fe4b-441e-9013-df25a120033c\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") "
Jul 22 02:07:11.060872 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060706    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8d551c-fe4b-441e-9013-df25a120033c-tls-certs\") pod \"2c8d551c-fe4b-441e-9013-df25a120033c\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") "
Jul 22 02:07:11.060872 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060777    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2f4s\" (UniqueName: \"kubernetes.io/projected/2c8d551c-fe4b-441e-9013-df25a120033c-kube-api-access-m2f4s\") pod \"2c8d551c-fe4b-441e-9013-df25a120033c\" (UID: \"2c8d551c-fe4b-441e-9013-df25a120033c\") "
Jul 22 02:07:11.061060 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060863    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c8d551c-fe4b-441e-9013-df25a120033c" (UID: "2c8d551c-fe4b-441e-9013-df25a120033c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:11.061060 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060875    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-home" (OuterVolumeSpecName: "home") pod "2c8d551c-fe4b-441e-9013-df25a120033c" (UID: "2c8d551c-fe4b-441e-9013-df25a120033c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:11.061060 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.060963    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c8d551c-fe4b-441e-9013-df25a120033c" (UID: "2c8d551c-fe4b-441e-9013-df25a120033c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:11.061173 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.061075    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:11.061173 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.061119    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:11.061173 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.061135    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:11.062981 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.062953    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8d551c-fe4b-441e-9013-df25a120033c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c8d551c-fe4b-441e-9013-df25a120033c" (UID: "2c8d551c-fe4b-441e-9013-df25a120033c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:07:11.063086 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.063018    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8d551c-fe4b-441e-9013-df25a120033c-kube-api-access-m2f4s" (OuterVolumeSpecName: "kube-api-access-m2f4s") pod "2c8d551c-fe4b-441e-9013-df25a120033c" (UID: "2c8d551c-fe4b-441e-9013-df25a120033c"). InnerVolumeSpecName "kube-api-access-m2f4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:07:11.063086 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.063059    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-dshm" (OuterVolumeSpecName: "dshm") pod "2c8d551c-fe4b-441e-9013-df25a120033c" (UID: "2c8d551c-fe4b-441e-9013-df25a120033c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:11.161494 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.161465    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8d551c-fe4b-441e-9013-df25a120033c-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:11.161494 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.161491    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8d551c-fe4b-441e-9013-df25a120033c-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:11.161658 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.161504    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2f4s\" (UniqueName: \"kubernetes.io/projected/2c8d551c-fe4b-441e-9013-df25a120033c-kube-api-access-m2f4s\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:11.879722 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.879687    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"
Jul 22 02:07:11.879722 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.879707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd" event={"ID":"2c8d551c-fe4b-441e-9013-df25a120033c","Type":"ContainerDied","Data":"1c1d22b1ca31a876205b9dfd6b0a4eeda599b77d333a8989ac84728acdc9a124"}
Jul 22 02:07:11.880265 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.879770    2561 scope.go:117] "RemoveContainer" containerID="c2d213f9dff697106c3565855185cd535e5ae125eac5329b1802675e498d118c"
Jul 22 02:07:11.911218 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.911188    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"]
Jul 22 02:07:11.920593 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:11.920567    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c838639d-kserve-f49787479-5kzfd"]
Jul 22 02:07:12.035045 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.035007    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr" podUID="d25df628-ed7b-48cf-8243-42956c07902c" containerName="main" containerID="cri-o://5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43" gracePeriod=45
Jul 22 02:07:12.285935 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.285911    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:07:12.372531 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372499    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-home\") pod \"d25df628-ed7b-48cf-8243-42956c07902c\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") "
Jul 22 02:07:12.372682 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-tmp-dir\") pod \"d25df628-ed7b-48cf-8243-42956c07902c\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") "
Jul 22 02:07:12.372682 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-model-cache\") pod \"d25df628-ed7b-48cf-8243-42956c07902c\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") "
Jul 22 02:07:12.372682 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372626    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkw2p\" (UniqueName: \"kubernetes.io/projected/d25df628-ed7b-48cf-8243-42956c07902c-kube-api-access-gkw2p\") pod \"d25df628-ed7b-48cf-8243-42956c07902c\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") "
Jul 22 02:07:12.372682 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372661    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-dshm\") pod \"d25df628-ed7b-48cf-8243-42956c07902c\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") "
Jul 22 02:07:12.372937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25df628-ed7b-48cf-8243-42956c07902c-tls-certs\") pod \"d25df628-ed7b-48cf-8243-42956c07902c\" (UID: \"d25df628-ed7b-48cf-8243-42956c07902c\") "
Jul 22 02:07:12.372937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-home" (OuterVolumeSpecName: "home") pod "d25df628-ed7b-48cf-8243-42956c07902c" (UID: "d25df628-ed7b-48cf-8243-42956c07902c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:12.372937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d25df628-ed7b-48cf-8243-42956c07902c" (UID: "d25df628-ed7b-48cf-8243-42956c07902c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:12.372937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.372902    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d25df628-ed7b-48cf-8243-42956c07902c" (UID: "d25df628-ed7b-48cf-8243-42956c07902c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:12.373109 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.373022    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:12.373109 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.373039    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:12.373109 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.373055    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:12.374890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.374865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d25df628-ed7b-48cf-8243-42956c07902c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d25df628-ed7b-48cf-8243-42956c07902c" (UID: "d25df628-ed7b-48cf-8243-42956c07902c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:07:12.374890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.374868    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-dshm" (OuterVolumeSpecName: "dshm") pod "d25df628-ed7b-48cf-8243-42956c07902c" (UID: "d25df628-ed7b-48cf-8243-42956c07902c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:07:12.374890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.374872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25df628-ed7b-48cf-8243-42956c07902c-kube-api-access-gkw2p" (OuterVolumeSpecName: "kube-api-access-gkw2p") pod "d25df628-ed7b-48cf-8243-42956c07902c" (UID: "d25df628-ed7b-48cf-8243-42956c07902c"). InnerVolumeSpecName "kube-api-access-gkw2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:07:12.474412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.474328    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkw2p\" (UniqueName: \"kubernetes.io/projected/d25df628-ed7b-48cf-8243-42956c07902c-kube-api-access-gkw2p\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:12.474412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.474359    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d25df628-ed7b-48cf-8243-42956c07902c-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:12.474412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.474368    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d25df628-ed7b-48cf-8243-42956c07902c-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:07:12.885921 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.885885    2561 generic.go:356] "Generic (PLEG): container finished" podID="d25df628-ed7b-48cf-8243-42956c07902c" containerID="5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43" exitCode=0
Jul 22 02:07:12.886361 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.885927    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"
Jul 22 02:07:12.886361 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.885967    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr" event={"ID":"d25df628-ed7b-48cf-8243-42956c07902c","Type":"ContainerDied","Data":"5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43"}
Jul 22 02:07:12.886361 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.886005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr" event={"ID":"d25df628-ed7b-48cf-8243-42956c07902c","Type":"ContainerDied","Data":"fbc3deebe336b5a316fe56dfabbb95bb7d566375b99de64bfeac4cb12f475ed6"}
Jul 22 02:07:12.886361 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.886024    2561 scope.go:117] "RemoveContainer" containerID="5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43"
Jul 22 02:07:12.895062 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.895048    2561 scope.go:117] "RemoveContainer" containerID="5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43"
Jul 22 02:07:12.895296 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:07:12.895275    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43\": container with ID starting with 5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43 not found: ID does not exist" containerID="5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43"
Jul 22 02:07:12.895351 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.895303    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43"} err="failed to get container status \"5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43\": rpc error: code = NotFound desc = could not find container \"5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43\": container with ID starting with 5ceca1a0134c95afb99a0df062ab3ae0e6248335656109679dd3d6ad6be82d43 not found: ID does not exist"
Jul 22 02:07:12.909648 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.909626    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"]
Jul 22 02:07:12.913335 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:12.913316    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c838639d-kserve-9cc7fbc-pgptr"]
Jul 22 02:07:13.102300 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:13.102266    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8d551c-fe4b-441e-9013-df25a120033c" path="/var/lib/kubelet/pods/2c8d551c-fe4b-441e-9013-df25a120033c/volumes"
Jul 22 02:07:13.102636 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:07:13.102624    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25df628-ed7b-48cf-8243-42956c07902c" path="/var/lib/kubelet/pods/d25df628-ed7b-48cf-8243-42956c07902c/volumes"
Jul 22 02:09:49.046522 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:09:49.046487    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:09:49.053762 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:09:49.053712    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:11:02.328506 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328469    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"]
Jul 22 02:11:02.329042 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328830    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25df628-ed7b-48cf-8243-42956c07902c" containerName="main"
Jul 22 02:11:02.329042 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328841    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25df628-ed7b-48cf-8243-42956c07902c" containerName="main"
Jul 22 02:11:02.329042 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8d551c-fe4b-441e-9013-df25a120033c" containerName="main"
Jul 22 02:11:02.329042 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328858    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8d551c-fe4b-441e-9013-df25a120033c" containerName="main"
Jul 22 02:11:02.329042 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328915    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8d551c-fe4b-441e-9013-df25a120033c" containerName="main"
Jul 22 02:11:02.329042 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.328923    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25df628-ed7b-48cf-8243-42956c07902c" containerName="main"
Jul 22 02:11:02.331854 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.331838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:02.334302 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.334279    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:11:02.339243 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.339045    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"]
Jul 22 02:11:02.390150 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.390107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfa9304-6417-448d-98b2-a882a12592a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a46a06be-96tcp\" (UID: \"ddfa9304-6417-448d-98b2-a882a12592a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:02.491048 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.491009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfa9304-6417-448d-98b2-a882a12592a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a46a06be-96tcp\" (UID: \"ddfa9304-6417-448d-98b2-a882a12592a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:02.491594 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.491576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfa9304-6417-448d-98b2-a882a12592a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a46a06be-96tcp\" (UID: \"ddfa9304-6417-448d-98b2-a882a12592a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:02.643010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.642921    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:02.774048 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.774004    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"]
Jul 22 02:11:02.776313 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:11:02.776285    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfa9304_6417_448d_98b2_a882a12592a7.slice/crio-bae85dca4d26e7f6bf469a2eb735a8796800a035133dfc6b8b6ce8f3037b28a4 WatchSource:0}: Error finding container bae85dca4d26e7f6bf469a2eb735a8796800a035133dfc6b8b6ce8f3037b28a4: Status 404 returned error can't find the container with id bae85dca4d26e7f6bf469a2eb735a8796800a035133dfc6b8b6ce8f3037b28a4
Jul 22 02:11:02.778149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.778135    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:11:02.903060 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:02.902983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp" event={"ID":"ddfa9304-6417-448d-98b2-a882a12592a7","Type":"ContainerStarted","Data":"bae85dca4d26e7f6bf469a2eb735a8796800a035133dfc6b8b6ce8f3037b28a4"}
Jul 22 02:11:10.945149 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:10.945117    2561 generic.go:356] "Generic (PLEG): container finished" podID="ddfa9304-6417-448d-98b2-a882a12592a7" containerID="065e63b78ed43859edf7c733a25427de9ad026fb89ddae377bfed905d43ab4d1" exitCode=0
Jul 22 02:11:10.945510 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:10.945211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp" event={"ID":"ddfa9304-6417-448d-98b2-a882a12592a7","Type":"ContainerDied","Data":"065e63b78ed43859edf7c733a25427de9ad026fb89ddae377bfed905d43ab4d1"}
Jul 22 02:11:12.078395 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.078369    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:12.176685 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.176655    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfa9304-6417-448d-98b2-a882a12592a7-maas-api-service-ca\") pod \"ddfa9304-6417-448d-98b2-a882a12592a7\" (UID: \"ddfa9304-6417-448d-98b2-a882a12592a7\") "
Jul 22 02:11:12.176998 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.176975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfa9304-6417-448d-98b2-a882a12592a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddfa9304-6417-448d-98b2-a882a12592a7" (UID: "ddfa9304-6417-448d-98b2-a882a12592a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:11:12.278112 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.278074    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddfa9304-6417-448d-98b2-a882a12592a7-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:11:12.955797 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.955761    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp" event={"ID":"ddfa9304-6417-448d-98b2-a882a12592a7","Type":"ContainerDied","Data":"bae85dca4d26e7f6bf469a2eb735a8796800a035133dfc6b8b6ce8f3037b28a4"}
Jul 22 02:11:12.955977 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.955813    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae85dca4d26e7f6bf469a2eb735a8796800a035133dfc6b8b6ce8f3037b28a4"
Jul 22 02:11:12.955977 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:12.955769    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"
Jul 22 02:11:27.425681 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:27.425642    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"]
Jul 22 02:11:27.428231 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:27.428211    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a46a06be-96tcp"]
Jul 22 02:11:29.101914 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:29.101886    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddfa9304-6417-448d-98b2-a882a12592a7" path="/var/lib/kubelet/pods/ddfa9304-6417-448d-98b2-a882a12592a7/volumes"
Jul 22 02:11:56.170945 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.170860    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"]
Jul 22 02:11:56.171432 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.171209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddfa9304-6417-448d-98b2-a882a12592a7" containerName="revoke-keys"
Jul 22 02:11:56.171432 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.171220    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddfa9304-6417-448d-98b2-a882a12592a7" containerName="revoke-keys"
Jul 22 02:11:56.171432 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.171277    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddfa9304-6417-448d-98b2-a882a12592a7" containerName="revoke-keys"
Jul 22 02:11:56.174226 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.174210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:11:56.176886 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.176867    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:11:56.183581 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.183547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"]
Jul 22 02:11:56.197419 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.197396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87aec257-cb6f-4042-aadc-372df3d6cc0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j\" (UID: \"87aec257-cb6f-4042-aadc-372df3d6cc0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:11:56.298136 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.298096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87aec257-cb6f-4042-aadc-372df3d6cc0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j\" (UID: \"87aec257-cb6f-4042-aadc-372df3d6cc0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:11:56.298723 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.298702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87aec257-cb6f-4042-aadc-372df3d6cc0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j\" (UID: \"87aec257-cb6f-4042-aadc-372df3d6cc0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:11:56.485639 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.485558    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:11:56.612785 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:56.612755    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"]
Jul 22 02:11:56.616335 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:11:56.616307    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87aec257_cb6f_4042_aadc_372df3d6cc0e.slice/crio-b81be610d35bad2377eb16d43d2a9b6582817981891cf2c1e9db8adfe6dad794 WatchSource:0}: Error finding container b81be610d35bad2377eb16d43d2a9b6582817981891cf2c1e9db8adfe6dad794: Status 404 returned error can't find the container with id b81be610d35bad2377eb16d43d2a9b6582817981891cf2c1e9db8adfe6dad794
Jul 22 02:11:57.147520 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:57.147429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j" event={"ID":"87aec257-cb6f-4042-aadc-372df3d6cc0e","Type":"ContainerStarted","Data":"86dd2861710ecc9de667ec7bd4f7972e48d2afd04eddaccd6877287606143b5e"}
Jul 22 02:11:57.147520 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:57.147487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j" event={"ID":"87aec257-cb6f-4042-aadc-372df3d6cc0e","Type":"ContainerStarted","Data":"b81be610d35bad2377eb16d43d2a9b6582817981891cf2c1e9db8adfe6dad794"}
Jul 22 02:11:57.177771 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:57.177696    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j" podStartSLOduration=1.177681976 podStartE2EDuration="1.177681976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:11:57.177659033 +0000 UTC m=+1628.680517275" watchObservedRunningTime="2026-07-22 02:11:57.177681976 +0000 UTC m=+1628.680540206"
Jul 22 02:11:58.153585 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:58.153542    2561 generic.go:356] "Generic (PLEG): container finished" podID="87aec257-cb6f-4042-aadc-372df3d6cc0e" containerID="86dd2861710ecc9de667ec7bd4f7972e48d2afd04eddaccd6877287606143b5e" exitCode=0
Jul 22 02:11:58.153784 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:58.153628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j" event={"ID":"87aec257-cb6f-4042-aadc-372df3d6cc0e","Type":"ContainerDied","Data":"86dd2861710ecc9de667ec7bd4f7972e48d2afd04eddaccd6877287606143b5e"}
Jul 22 02:11:59.289703 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:59.289680    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:11:59.325746 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:59.325695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87aec257-cb6f-4042-aadc-372df3d6cc0e-maas-api-service-ca\") pod \"87aec257-cb6f-4042-aadc-372df3d6cc0e\" (UID: \"87aec257-cb6f-4042-aadc-372df3d6cc0e\") "
Jul 22 02:11:59.326027 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:59.326002    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87aec257-cb6f-4042-aadc-372df3d6cc0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87aec257-cb6f-4042-aadc-372df3d6cc0e" (UID: "87aec257-cb6f-4042-aadc-372df3d6cc0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:11:59.426847 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:11:59.426780    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87aec257-cb6f-4042-aadc-372df3d6cc0e-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:12:00.164268 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:00.164232    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"
Jul 22 02:12:00.164421 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:00.164235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j" event={"ID":"87aec257-cb6f-4042-aadc-372df3d6cc0e","Type":"ContainerDied","Data":"b81be610d35bad2377eb16d43d2a9b6582817981891cf2c1e9db8adfe6dad794"}
Jul 22 02:12:00.164421 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:00.164353    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b81be610d35bad2377eb16d43d2a9b6582817981891cf2c1e9db8adfe6dad794"
Jul 22 02:12:11.270269 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:11.270229    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"]
Jul 22 02:12:11.272583 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:11.272525    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1e59c61a-dhl8j"]
Jul 22 02:12:13.102611 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:13.102575    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87aec257-cb6f-4042-aadc-372df3d6cc0e" path="/var/lib/kubelet/pods/87aec257-cb6f-4042-aadc-372df3d6cc0e/volumes"
Jul 22 02:12:15.309679 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.309642    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"]
Jul 22 02:12:15.310076 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.309996    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aec257-cb6f-4042-aadc-372df3d6cc0e" containerName="revoke-keys"
Jul 22 02:12:15.310076 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.310009    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aec257-cb6f-4042-aadc-372df3d6cc0e" containerName="revoke-keys"
Jul 22 02:12:15.310076 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.310076    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87aec257-cb6f-4042-aadc-372df3d6cc0e" containerName="revoke-keys"
Jul 22 02:12:15.313179 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.313162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.315708 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.315685    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:12:15.315870 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.315723    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-68767aef-serving-cert\""
Jul 22 02:12:15.323027 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.322993    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"]
Jul 22 02:12:15.352108 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.352067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec995d4b-558d-463c-ace5-23d02ec01b07-openshift-service-ca\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.352284 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.352179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwt7\" (UniqueName: \"kubernetes.io/projected/ec995d4b-558d-463c-ace5-23d02ec01b07-kube-api-access-xrwt7\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.352284 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.352223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec995d4b-558d-463c-ace5-23d02ec01b07-maas-api-tls\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.452925 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.452892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec995d4b-558d-463c-ace5-23d02ec01b07-openshift-service-ca\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.453116 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.452955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwt7\" (UniqueName: \"kubernetes.io/projected/ec995d4b-558d-463c-ace5-23d02ec01b07-kube-api-access-xrwt7\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.453116 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.452997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec995d4b-558d-463c-ace5-23d02ec01b07-maas-api-tls\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.453589 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.453566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec995d4b-558d-463c-ace5-23d02ec01b07-openshift-service-ca\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.455467 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.455444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec995d4b-558d-463c-ace5-23d02ec01b07-maas-api-tls\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.463753 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.463720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwt7\" (UniqueName: \"kubernetes.io/projected/ec995d4b-558d-463c-ace5-23d02ec01b07-kube-api-access-xrwt7\") pod \"maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.623714 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.623633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:15.760404 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:15.760376    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"]
Jul 22 02:12:15.762809 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:12:15.762780    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec995d4b_558d_463c_ace5_23d02ec01b07.slice/crio-92ff2630fde6f9ffbd927a3114f1cc8777090dd9f479ef088ab6e2e1f9da342e WatchSource:0}: Error finding container 92ff2630fde6f9ffbd927a3114f1cc8777090dd9f479ef088ab6e2e1f9da342e: Status 404 returned error can't find the container with id 92ff2630fde6f9ffbd927a3114f1cc8777090dd9f479ef088ab6e2e1f9da342e
Jul 22 02:12:16.235660 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:16.235624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt" event={"ID":"ec995d4b-558d-463c-ace5-23d02ec01b07","Type":"ContainerStarted","Data":"92ff2630fde6f9ffbd927a3114f1cc8777090dd9f479ef088ab6e2e1f9da342e"}
Jul 22 02:12:18.246650 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:18.246613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt" event={"ID":"ec995d4b-558d-463c-ace5-23d02ec01b07","Type":"ContainerStarted","Data":"3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605"}
Jul 22 02:12:18.247133 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:18.246816    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:18.266303 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:18.266261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt" podStartSLOduration=1.703173789 podStartE2EDuration="3.266247058s" totalImagesPullingTime="1.563073269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:12:18.264618761 +0000 UTC m=+1649.767476990" watchObservedRunningTime="2026-07-22 02:12:18.266247058 +0000 UTC m=+1649.769105304"
Jul 22 02:12:24.256128 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:24.256092    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:41.271997 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.271963    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"]
Jul 22 02:12:41.275334 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.275317    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:41.277951 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.277932    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:12:41.282661 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.282642    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"]
Jul 22 02:12:41.365890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.365857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cae84c-1296-47a1-9fa2-fcd53658dc4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-68767aef-n7tzm\" (UID: \"86cae84c-1296-47a1-9fa2-fcd53658dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:41.467283 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.467249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cae84c-1296-47a1-9fa2-fcd53658dc4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-68767aef-n7tzm\" (UID: \"86cae84c-1296-47a1-9fa2-fcd53658dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:41.467885 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.467865    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cae84c-1296-47a1-9fa2-fcd53658dc4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-68767aef-n7tzm\" (UID: \"86cae84c-1296-47a1-9fa2-fcd53658dc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:41.586533 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.586469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:41.713510 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:41.713430    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"]
Jul 22 02:12:41.715854 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:12:41.715828    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cae84c_1296_47a1_9fa2_fcd53658dc4d.slice/crio-60c95f8be79cdad048e53c7cb21a2998871e27d3a55be2b79c38010b3e697dd8 WatchSource:0}: Error finding container 60c95f8be79cdad048e53c7cb21a2998871e27d3a55be2b79c38010b3e697dd8: Status 404 returned error can't find the container with id 60c95f8be79cdad048e53c7cb21a2998871e27d3a55be2b79c38010b3e697dd8
Jul 22 02:12:42.359819 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:42.359787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm" event={"ID":"86cae84c-1296-47a1-9fa2-fcd53658dc4d","Type":"ContainerStarted","Data":"2f9ea1c53076b0435dfc529a0479ebeddefe03f992f2e5bbe8e53652cc92c682"}
Jul 22 02:12:42.360171 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:42.359843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm" event={"ID":"86cae84c-1296-47a1-9fa2-fcd53658dc4d","Type":"ContainerStarted","Data":"60c95f8be79cdad048e53c7cb21a2998871e27d3a55be2b79c38010b3e697dd8"}
Jul 22 02:12:42.376514 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:42.376465    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm" podStartSLOduration=1.376450153 podStartE2EDuration="1.376450153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:12:42.374464026 +0000 UTC m=+1673.877322270" watchObservedRunningTime="2026-07-22 02:12:42.376450153 +0000 UTC m=+1673.879308381"
Jul 22 02:12:43.365592 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:43.365563    2561 generic.go:356] "Generic (PLEG): container finished" podID="86cae84c-1296-47a1-9fa2-fcd53658dc4d" containerID="2f9ea1c53076b0435dfc529a0479ebeddefe03f992f2e5bbe8e53652cc92c682" exitCode=0
Jul 22 02:12:43.365993 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:43.365629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm" event={"ID":"86cae84c-1296-47a1-9fa2-fcd53658dc4d","Type":"ContainerDied","Data":"2f9ea1c53076b0435dfc529a0479ebeddefe03f992f2e5bbe8e53652cc92c682"}
Jul 22 02:12:44.506832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:44.506807    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:44.589265 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:44.589239    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cae84c-1296-47a1-9fa2-fcd53658dc4d-maas-api-service-ca\") pod \"86cae84c-1296-47a1-9fa2-fcd53658dc4d\" (UID: \"86cae84c-1296-47a1-9fa2-fcd53658dc4d\") "
Jul 22 02:12:44.589562 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:44.589539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cae84c-1296-47a1-9fa2-fcd53658dc4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86cae84c-1296-47a1-9fa2-fcd53658dc4d" (UID: "86cae84c-1296-47a1-9fa2-fcd53658dc4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:12:44.690589 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:44.690507    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86cae84c-1296-47a1-9fa2-fcd53658dc4d-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:12:45.378122 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:45.378046    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"
Jul 22 02:12:45.378122 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:45.378093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm" event={"ID":"86cae84c-1296-47a1-9fa2-fcd53658dc4d","Type":"ContainerDied","Data":"60c95f8be79cdad048e53c7cb21a2998871e27d3a55be2b79c38010b3e697dd8"}
Jul 22 02:12:45.378303 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:45.378137    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60c95f8be79cdad048e53c7cb21a2998871e27d3a55be2b79c38010b3e697dd8"
Jul 22 02:12:51.947523 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:51.947490    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"]
Jul 22 02:12:51.948039 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:51.947718    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt" podUID="ec995d4b-558d-463c-ace5-23d02ec01b07" containerName="maas-api" containerID="cri-o://3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605" gracePeriod=30
Jul 22 02:12:52.196046 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.196022    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:52.349814 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.349779    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec995d4b-558d-463c-ace5-23d02ec01b07-maas-api-tls\") pod \"ec995d4b-558d-463c-ace5-23d02ec01b07\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") "
Jul 22 02:12:52.350000 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.349832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrwt7\" (UniqueName: \"kubernetes.io/projected/ec995d4b-558d-463c-ace5-23d02ec01b07-kube-api-access-xrwt7\") pod \"ec995d4b-558d-463c-ace5-23d02ec01b07\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") "
Jul 22 02:12:52.350000 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.349868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec995d4b-558d-463c-ace5-23d02ec01b07-openshift-service-ca\") pod \"ec995d4b-558d-463c-ace5-23d02ec01b07\" (UID: \"ec995d4b-558d-463c-ace5-23d02ec01b07\") "
Jul 22 02:12:52.350278 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.350253    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec995d4b-558d-463c-ace5-23d02ec01b07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec995d4b-558d-463c-ace5-23d02ec01b07" (UID: "ec995d4b-558d-463c-ace5-23d02ec01b07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:12:52.351963 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.351928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec995d4b-558d-463c-ace5-23d02ec01b07-kube-api-access-xrwt7" (OuterVolumeSpecName: "kube-api-access-xrwt7") pod "ec995d4b-558d-463c-ace5-23d02ec01b07" (UID: "ec995d4b-558d-463c-ace5-23d02ec01b07"). InnerVolumeSpecName "kube-api-access-xrwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:12:52.352075 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.351967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec995d4b-558d-463c-ace5-23d02ec01b07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec995d4b-558d-463c-ace5-23d02ec01b07" (UID: "ec995d4b-558d-463c-ace5-23d02ec01b07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:12:52.412414 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.412390    2561 generic.go:356] "Generic (PLEG): container finished" podID="ec995d4b-558d-463c-ace5-23d02ec01b07" containerID="3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605" exitCode=0
Jul 22 02:12:52.412546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.412433    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"
Jul 22 02:12:52.412546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.412484    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt" event={"ID":"ec995d4b-558d-463c-ace5-23d02ec01b07","Type":"ContainerDied","Data":"3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605"}
Jul 22 02:12:52.412546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.412538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt" event={"ID":"ec995d4b-558d-463c-ace5-23d02ec01b07","Type":"ContainerDied","Data":"92ff2630fde6f9ffbd927a3114f1cc8777090dd9f479ef088ab6e2e1f9da342e"}
Jul 22 02:12:52.412663 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.412563    2561 scope.go:117] "RemoveContainer" containerID="3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605"
Jul 22 02:12:52.420896 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.420877    2561 scope.go:117] "RemoveContainer" containerID="3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605"
Jul 22 02:12:52.421140 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:12:52.421123    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605\": container with ID starting with 3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605 not found: ID does not exist" containerID="3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605"
Jul 22 02:12:52.421180 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.421150    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605"} err="failed to get container status \"3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605\": rpc error: code = NotFound desc = could not find container \"3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605\": container with ID starting with 3c26dec159e61fef99b80b7dfe2b58f5911b83a552ce98ec54bd99d71dc65605 not found: ID does not exist"
Jul 22 02:12:52.435345 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.435321    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"]
Jul 22 02:12:52.439615 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.439588    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-68767aef-5f5c7475fd-d8gbt"]
Jul 22 02:12:52.451047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.451025    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec995d4b-558d-463c-ace5-23d02ec01b07-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:12:52.451047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.451044    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwt7\" (UniqueName: \"kubernetes.io/projected/ec995d4b-558d-463c-ace5-23d02ec01b07-kube-api-access-xrwt7\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:12:52.451164 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:52.451054    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec995d4b-558d-463c-ace5-23d02ec01b07-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:12:53.102695 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:53.102662    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec995d4b-558d-463c-ace5-23d02ec01b07" path="/var/lib/kubelet/pods/ec995d4b-558d-463c-ace5-23d02ec01b07/volumes"
Jul 22 02:12:56.438621 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:56.438592    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"]
Jul 22 02:12:56.445121 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:56.445089    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-68767aef-n7tzm"]
Jul 22 02:12:57.101803 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:12:57.101773    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86cae84c-1296-47a1-9fa2-fcd53658dc4d" path="/var/lib/kubelet/pods/86cae84c-1296-47a1-9fa2-fcd53658dc4d/volumes"
Jul 22 02:13:01.586862 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.586824    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"]
Jul 22 02:13:01.587376 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.587357    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86cae84c-1296-47a1-9fa2-fcd53658dc4d" containerName="revoke-keys"
Jul 22 02:13:01.587465 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.587378    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="86cae84c-1296-47a1-9fa2-fcd53658dc4d" containerName="revoke-keys"
Jul 22 02:13:01.587465 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.587398    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec995d4b-558d-463c-ace5-23d02ec01b07" containerName="maas-api"
Jul 22 02:13:01.587465 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.587407    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec995d4b-558d-463c-ace5-23d02ec01b07" containerName="maas-api"
Jul 22 02:13:01.587584 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.587501    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="86cae84c-1296-47a1-9fa2-fcd53658dc4d" containerName="revoke-keys"
Jul 22 02:13:01.587584 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.587512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec995d4b-558d-463c-ace5-23d02ec01b07" containerName="maas-api"
Jul 22 02:13:01.590600 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.590584    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.593520 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.593499    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4735213f-serving-cert\""
Jul 22 02:13:01.593624 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.593534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:13:01.599341 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.599317    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"]
Jul 22 02:13:01.633345 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.633310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvcw\" (UniqueName: \"kubernetes.io/projected/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-kube-api-access-hwvcw\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.633514 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.633369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.633514 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.633407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-openshift-service-ca\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.733878 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.733843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvcw\" (UniqueName: \"kubernetes.io/projected/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-kube-api-access-hwvcw\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.734070 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.733899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.734070 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.733940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-openshift-service-ca\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.734199 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:13:01.734085    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-serving-cert: secret "maas-api-e2e-derive-4735213f-serving-cert" not found
Jul 22 02:13:01.734199 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:13:01.734180    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls podName:9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce nodeName:}" failed. No retries permitted until 2026-07-22 02:13:02.234157985 +0000 UTC m=+1693.737016202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls") pod "maas-api-e2e-derive-4735213f-98448c79d-7vtct" (UID: "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce") : secret "maas-api-e2e-derive-4735213f-serving-cert" not found
Jul 22 02:13:01.734622 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.734601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-openshift-service-ca\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:01.745380 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:01.745353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvcw\" (UniqueName: \"kubernetes.io/projected/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-kube-api-access-hwvcw\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:02.238112 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:02.238073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:02.240664 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:02.240639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls\") pod \"maas-api-e2e-derive-4735213f-98448c79d-7vtct\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:02.500963 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:02.500873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:02.637686 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:02.637468    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"]
Jul 22 02:13:02.640430 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:13:02.640389    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9773af75_4a5b_4bf9_87ee_e1eb29c3c0ce.slice/crio-45bad3727cd5239e58cb5e2a6d41fcdb2820c397733a3c92cb28ba7da37de1bb WatchSource:0}: Error finding container 45bad3727cd5239e58cb5e2a6d41fcdb2820c397733a3c92cb28ba7da37de1bb: Status 404 returned error can't find the container with id 45bad3727cd5239e58cb5e2a6d41fcdb2820c397733a3c92cb28ba7da37de1bb
Jul 22 02:13:03.463201 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:03.463163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct" event={"ID":"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce","Type":"ContainerStarted","Data":"f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4"}
Jul 22 02:13:03.463382 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:03.463225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct" event={"ID":"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce","Type":"ContainerStarted","Data":"45bad3727cd5239e58cb5e2a6d41fcdb2820c397733a3c92cb28ba7da37de1bb"}
Jul 22 02:13:03.463382 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:03.463252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:03.484375 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:03.484322    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct" podStartSLOduration=2.128545748 podStartE2EDuration="2.484306384s" totalImagesPullingTime="355.760636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:13:03.482250032 +0000 UTC m=+1694.985108272" watchObservedRunningTime="2026-07-22 02:13:03.484306384 +0000 UTC m=+1694.987164613"
Jul 22 02:13:09.473883 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:09.473849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:26.484259 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.484181    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"]
Jul 22 02:13:26.487661 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.487644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:26.490381 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.490361    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:13:26.496613 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.496583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"]
Jul 22 02:13:26.541012 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.540984    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2957717b-1041-4e45-8b94-9c84294e69cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4735213f-c6dh6\" (UID: \"2957717b-1041-4e45-8b94-9c84294e69cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:26.642389 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.642358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2957717b-1041-4e45-8b94-9c84294e69cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4735213f-c6dh6\" (UID: \"2957717b-1041-4e45-8b94-9c84294e69cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:26.643014 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.642996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2957717b-1041-4e45-8b94-9c84294e69cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4735213f-c6dh6\" (UID: \"2957717b-1041-4e45-8b94-9c84294e69cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:26.798920 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:26.798891    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:27.142746 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:27.142704    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"]
Jul 22 02:13:27.145090 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:13:27.145064    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2957717b_1041_4e45_8b94_9c84294e69cf.slice/crio-8b46c2ae588b62eaab803d7b7163e44b77c2823705fa424abdf952189df6d30a WatchSource:0}: Error finding container 8b46c2ae588b62eaab803d7b7163e44b77c2823705fa424abdf952189df6d30a: Status 404 returned error can't find the container with id 8b46c2ae588b62eaab803d7b7163e44b77c2823705fa424abdf952189df6d30a
Jul 22 02:13:27.576570 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:27.576538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6" event={"ID":"2957717b-1041-4e45-8b94-9c84294e69cf","Type":"ContainerStarted","Data":"bc2a3facc5edb29df6a0d0d62bcc4261d67c9a89b236e22701896d78f238b2f7"}
Jul 22 02:13:27.576981 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:27.576588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6" event={"ID":"2957717b-1041-4e45-8b94-9c84294e69cf","Type":"ContainerStarted","Data":"8b46c2ae588b62eaab803d7b7163e44b77c2823705fa424abdf952189df6d30a"}
Jul 22 02:13:27.598782 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:27.598717    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6" podStartSLOduration=1.598702543 podStartE2EDuration="1.598702543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:13:27.597750079 +0000 UTC m=+1719.100608300" watchObservedRunningTime="2026-07-22 02:13:27.598702543 +0000 UTC m=+1719.101560772"
Jul 22 02:13:28.582494 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:28.582458    2561 generic.go:356] "Generic (PLEG): container finished" podID="2957717b-1041-4e45-8b94-9c84294e69cf" containerID="bc2a3facc5edb29df6a0d0d62bcc4261d67c9a89b236e22701896d78f238b2f7" exitCode=0
Jul 22 02:13:28.582908 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:28.582550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6" event={"ID":"2957717b-1041-4e45-8b94-9c84294e69cf","Type":"ContainerDied","Data":"bc2a3facc5edb29df6a0d0d62bcc4261d67c9a89b236e22701896d78f238b2f7"}
Jul 22 02:13:29.719485 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:29.719459    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:29.767674 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:29.767650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2957717b-1041-4e45-8b94-9c84294e69cf-maas-api-service-ca\") pod \"2957717b-1041-4e45-8b94-9c84294e69cf\" (UID: \"2957717b-1041-4e45-8b94-9c84294e69cf\") "
Jul 22 02:13:29.768003 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:29.767979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2957717b-1041-4e45-8b94-9c84294e69cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2957717b-1041-4e45-8b94-9c84294e69cf" (UID: "2957717b-1041-4e45-8b94-9c84294e69cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:13:29.868984 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:29.868921    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2957717b-1041-4e45-8b94-9c84294e69cf-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:13:30.593519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:30.593495    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"
Jul 22 02:13:30.593519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:30.593514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6" event={"ID":"2957717b-1041-4e45-8b94-9c84294e69cf","Type":"ContainerDied","Data":"8b46c2ae588b62eaab803d7b7163e44b77c2823705fa424abdf952189df6d30a"}
Jul 22 02:13:30.593759 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:30.593558    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b46c2ae588b62eaab803d7b7163e44b77c2823705fa424abdf952189df6d30a"
Jul 22 02:13:37.021306 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.021265    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"]
Jul 22 02:13:37.021890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.021571    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct" podUID="9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" containerName="maas-api" containerID="cri-o://f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4" gracePeriod=30
Jul 22 02:13:37.274873 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.274814    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:37.429402 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.429370    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-openshift-service-ca\") pod \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") "
Jul 22 02:13:37.429552 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.429426    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwvcw\" (UniqueName: \"kubernetes.io/projected/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-kube-api-access-hwvcw\") pod \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") "
Jul 22 02:13:37.429552 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.429475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls\") pod \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\" (UID: \"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce\") "
Jul 22 02:13:37.429798 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.429770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" (UID: "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:13:37.431484 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.431453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" (UID: "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:13:37.431576 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.431496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-kube-api-access-hwvcw" (OuterVolumeSpecName: "kube-api-access-hwvcw") pod "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" (UID: "9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce"). InnerVolumeSpecName "kube-api-access-hwvcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:13:37.531012 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.530944    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvcw\" (UniqueName: \"kubernetes.io/projected/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-kube-api-access-hwvcw\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:13:37.531012 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.530973    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:13:37.531012 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.530984    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:13:37.630902 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.630872    2561 generic.go:356] "Generic (PLEG): container finished" podID="9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" containerID="f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4" exitCode=0
Jul 22 02:13:37.631061 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.630913    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"
Jul 22 02:13:37.631061 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.630933    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct" event={"ID":"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce","Type":"ContainerDied","Data":"f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4"}
Jul 22 02:13:37.631061 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.630981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct" event={"ID":"9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce","Type":"ContainerDied","Data":"45bad3727cd5239e58cb5e2a6d41fcdb2820c397733a3c92cb28ba7da37de1bb"}
Jul 22 02:13:37.631061 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.631007    2561 scope.go:117] "RemoveContainer" containerID="f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4"
Jul 22 02:13:37.640675 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.640650    2561 scope.go:117] "RemoveContainer" containerID="f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4"
Jul 22 02:13:37.640967 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:13:37.640943    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4\": container with ID starting with f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4 not found: ID does not exist" containerID="f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4"
Jul 22 02:13:37.641068 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.640974    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4"} err="failed to get container status \"f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4\": rpc error: code = NotFound desc = could not find container \"f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4\": container with ID starting with f27e4b8fd6dd09be831d243be5e0bebcdcafabf09fc66e15fa8ca9ab207a13d4 not found: ID does not exist"
Jul 22 02:13:37.659687 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.659661    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"]
Jul 22 02:13:37.663107 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:37.663085    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4735213f-98448c79d-7vtct"]
Jul 22 02:13:39.101824 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:39.101791    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" path="/var/lib/kubelet/pods/9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce/volumes"
Jul 22 02:13:41.587162 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:41.587133    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"]
Jul 22 02:13:41.589926 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:41.589902    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4735213f-c6dh6"]
Jul 22 02:13:43.102287 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:13:43.102254    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2957717b-1041-4e45-8b94-9c84294e69cf" path="/var/lib/kubelet/pods/2957717b-1041-4e45-8b94-9c84294e69cf/volumes"
Jul 22 02:14:49.084498 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:14:49.084471    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:14:49.086219 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:14:49.086188    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:15:00.141879 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.141845    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"]
Jul 22 02:15:00.142246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.142169    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2957717b-1041-4e45-8b94-9c84294e69cf" containerName="revoke-keys"
Jul 22 02:15:00.142246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.142179    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2957717b-1041-4e45-8b94-9c84294e69cf" containerName="revoke-keys"
Jul 22 02:15:00.142246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.142198    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" containerName="maas-api"
Jul 22 02:15:00.142246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.142204    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" containerName="maas-api"
Jul 22 02:15:00.142380 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.142254    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2957717b-1041-4e45-8b94-9c84294e69cf" containerName="revoke-keys"
Jul 22 02:15:00.142380 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.142262    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9773af75-4a5b-4bf9-87ee-e1eb29c3c0ce" containerName="maas-api"
Jul 22 02:15:00.145193 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.145177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"
Jul 22 02:15:00.147669 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.147647    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:15:00.153923 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.153886    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"]
Jul 22 02:15:00.157332 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.157316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"
Jul 22 02:15:00.288342 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:00.288286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"]
Jul 22 02:15:00.290533 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:15:00.290510    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a4dc39_3572_4dcf_a4f3_d46c0ba4dccb.slice/crio-1c518a0c1701a7f0c5b4e71dee4f0a76d61758979a830221ed772f53eb274d05 WatchSource:0}: Error finding container 1c518a0c1701a7f0c5b4e71dee4f0a76d61758979a830221ed772f53eb274d05: Status 404 returned error can't find the container with id 1c518a0c1701a7f0c5b4e71dee4f0a76d61758979a830221ed772f53eb274d05
Jul 22 02:15:01.008370 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:01.008333    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w" event={"ID":"16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb","Type":"ContainerStarted","Data":"765ad9d6e7957161e6f5799104cb7e10b3985cef0eaad5d5ffc6563b7bd682e5"}
Jul 22 02:15:01.008586 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:01.008384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w" event={"ID":"16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb","Type":"ContainerStarted","Data":"1c518a0c1701a7f0c5b4e71dee4f0a76d61758979a830221ed772f53eb274d05"}
Jul 22 02:15:01.024992 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:01.024939    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w" podStartSLOduration=1.024925201 podStartE2EDuration="1.024925201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:01.023166624 +0000 UTC m=+1812.526024854" watchObservedRunningTime="2026-07-22 02:15:01.024925201 +0000 UTC m=+1812.527783430"
Jul 22 02:15:02.014881 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:02.014847    2561 generic.go:356] "Generic (PLEG): container finished" podID="16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb" containerID="765ad9d6e7957161e6f5799104cb7e10b3985cef0eaad5d5ffc6563b7bd682e5" exitCode=0
Jul 22 02:15:02.015266 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:02.014900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w" event={"ID":"16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb","Type":"ContainerDied","Data":"765ad9d6e7957161e6f5799104cb7e10b3985cef0eaad5d5ffc6563b7bd682e5"}
Jul 22 02:15:03.154485 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:03.154460    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"
Jul 22 02:15:04.025894 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:04.025865    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w"
Jul 22 02:15:04.025894 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:04.025877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744775-xxn5w" event={"ID":"16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb","Type":"ContainerDied","Data":"1c518a0c1701a7f0c5b4e71dee4f0a76d61758979a830221ed772f53eb274d05"}
Jul 22 02:15:04.026154 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:04.025934    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c518a0c1701a7f0c5b4e71dee4f0a76d61758979a830221ed772f53eb274d05"
Jul 22 02:15:31.398040 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.398003    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"]
Jul 22 02:15:31.398879 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.398515    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb" containerName="cleanup"
Jul 22 02:15:31.398879 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.398536    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb" containerName="cleanup"
Jul 22 02:15:31.398879 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.398620    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb" containerName="cleanup"
Jul 22 02:15:31.401905 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.401885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.404854 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.404832    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:15:31.404953 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.404881    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4e4fadaa-serving-cert\""
Jul 22 02:15:31.414330 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.414304    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"]
Jul 22 02:15:31.480938 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.480908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.481126 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.480944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdtx\" (UniqueName: \"kubernetes.io/projected/352e888c-4769-49f6-92be-f1b906c64832-kube-api-access-qrdtx\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.481126 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.481046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352e888c-4769-49f6-92be-f1b906c64832-openshift-service-ca\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.581796 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.581761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdtx\" (UniqueName: \"kubernetes.io/projected/352e888c-4769-49f6-92be-f1b906c64832-kube-api-access-qrdtx\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.581969 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.581851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352e888c-4769-49f6-92be-f1b906c64832-openshift-service-ca\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.581969 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.581887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.582057 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:15:31.582000    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-serving-cert: secret "maas-api-e2e-mt-4e4fadaa-serving-cert" not found
Jul 22 02:15:31.582101 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:15:31.582065    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls podName:352e888c-4769-49f6-92be-f1b906c64832 nodeName:}" failed. No retries permitted until 2026-07-22 02:15:32.082044189 +0000 UTC m=+1843.584902396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls") pod "maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" (UID: "352e888c-4769-49f6-92be-f1b906c64832") : secret "maas-api-e2e-mt-4e4fadaa-serving-cert" not found
Jul 22 02:15:31.582566 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.582546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352e888c-4769-49f6-92be-f1b906c64832-openshift-service-ca\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:31.600905 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:31.600872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdtx\" (UniqueName: \"kubernetes.io/projected/352e888c-4769-49f6-92be-f1b906c64832-kube-api-access-qrdtx\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:32.086135 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:32.086103    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:32.088527 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:32.088503    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls\") pod \"maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:32.312406 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:32.312370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:32.446793 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:32.446767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"]
Jul 22 02:15:32.449004 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:15:32.448975    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352e888c_4769_49f6_92be_f1b906c64832.slice/crio-eab10de01838fdde7c104a5225073e5bd6016f8566cd842c16cc329709cb92ea WatchSource:0}: Error finding container eab10de01838fdde7c104a5225073e5bd6016f8566cd842c16cc329709cb92ea: Status 404 returned error can't find the container with id eab10de01838fdde7c104a5225073e5bd6016f8566cd842c16cc329709cb92ea
Jul 22 02:15:33.166502 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:33.166411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" event={"ID":"352e888c-4769-49f6-92be-f1b906c64832","Type":"ContainerStarted","Data":"6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69"}
Jul 22 02:15:33.166502 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:33.166476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" event={"ID":"352e888c-4769-49f6-92be-f1b906c64832","Type":"ContainerStarted","Data":"eab10de01838fdde7c104a5225073e5bd6016f8566cd842c16cc329709cb92ea"}
Jul 22 02:15:33.166676 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:33.166582    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:33.186517 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:33.186474    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" podStartSLOduration=1.8040984230000001 podStartE2EDuration="2.186460532s" totalImagesPullingTime="382.362109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:33.184856209 +0000 UTC m=+1844.687714476" watchObservedRunningTime="2026-07-22 02:15:33.186460532 +0000 UTC m=+1844.689318796"
Jul 22 02:15:39.176525 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:39.176458    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:43.358884 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.358848    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"]
Jul 22 02:15:43.362237 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.362221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:43.364890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.364869    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:15:43.369709 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.369672    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"]
Jul 22 02:15:43.479525 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.479490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ddcc-546b-4df7-a6c6-c17e96c069f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj\" (UID: \"1c63ddcc-546b-4df7-a6c6-c17e96c069f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:43.580844 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.580802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ddcc-546b-4df7-a6c6-c17e96c069f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj\" (UID: \"1c63ddcc-546b-4df7-a6c6-c17e96c069f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:43.581463 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.581442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ddcc-546b-4df7-a6c6-c17e96c069f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj\" (UID: \"1c63ddcc-546b-4df7-a6c6-c17e96c069f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:43.674171 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.674091    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:43.817953 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:43.817928    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"]
Jul 22 02:15:43.820239 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:15:43.820209    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c63ddcc_546b_4df7_a6c6_c17e96c069f8.slice/crio-92059d8926d5a113fd48ee1882775b93fec10e0f5487fced0f91490fb5998121 WatchSource:0}: Error finding container 92059d8926d5a113fd48ee1882775b93fec10e0f5487fced0f91490fb5998121: Status 404 returned error can't find the container with id 92059d8926d5a113fd48ee1882775b93fec10e0f5487fced0f91490fb5998121
Jul 22 02:15:44.218630 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:44.218600    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj" event={"ID":"1c63ddcc-546b-4df7-a6c6-c17e96c069f8","Type":"ContainerStarted","Data":"97f51f87d3838b096b5238238f43bc42bff8b92b140cb01682a94fbd937e6238"}
Jul 22 02:15:44.218777 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:44.218653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj" event={"ID":"1c63ddcc-546b-4df7-a6c6-c17e96c069f8","Type":"ContainerStarted","Data":"92059d8926d5a113fd48ee1882775b93fec10e0f5487fced0f91490fb5998121"}
Jul 22 02:15:44.233676 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:44.233631    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj" podStartSLOduration=1.233616969 podStartE2EDuration="1.233616969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:15:44.23267066 +0000 UTC m=+1855.735528889" watchObservedRunningTime="2026-07-22 02:15:44.233616969 +0000 UTC m=+1855.736475198"
Jul 22 02:15:45.224386 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:45.224353    2561 generic.go:356] "Generic (PLEG): container finished" podID="1c63ddcc-546b-4df7-a6c6-c17e96c069f8" containerID="97f51f87d3838b096b5238238f43bc42bff8b92b140cb01682a94fbd937e6238" exitCode=0
Jul 22 02:15:45.224780 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:45.224395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj" event={"ID":"1c63ddcc-546b-4df7-a6c6-c17e96c069f8","Type":"ContainerDied","Data":"97f51f87d3838b096b5238238f43bc42bff8b92b140cb01682a94fbd937e6238"}
Jul 22 02:15:46.361102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:46.361080    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:46.505313 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:46.505285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ddcc-546b-4df7-a6c6-c17e96c069f8-maas-api-service-ca\") pod \"1c63ddcc-546b-4df7-a6c6-c17e96c069f8\" (UID: \"1c63ddcc-546b-4df7-a6c6-c17e96c069f8\") "
Jul 22 02:15:46.505656 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:46.505633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c63ddcc-546b-4df7-a6c6-c17e96c069f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c63ddcc-546b-4df7-a6c6-c17e96c069f8" (UID: "1c63ddcc-546b-4df7-a6c6-c17e96c069f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:15:46.606341 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:46.606317    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c63ddcc-546b-4df7-a6c6-c17e96c069f8-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:15:47.236282 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:47.236253    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"
Jul 22 02:15:47.236467 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:47.236254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj" event={"ID":"1c63ddcc-546b-4df7-a6c6-c17e96c069f8","Type":"ContainerDied","Data":"92059d8926d5a113fd48ee1882775b93fec10e0f5487fced0f91490fb5998121"}
Jul 22 02:15:47.236467 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:47.236401    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92059d8926d5a113fd48ee1882775b93fec10e0f5487fced0f91490fb5998121"
Jul 22 02:15:53.941883 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:53.941853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"]
Jul 22 02:15:53.942229 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:53.942109    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" podUID="352e888c-4769-49f6-92be-f1b906c64832" containerName="maas-api" containerID="cri-o://6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69" gracePeriod=30
Jul 22 02:15:54.188864 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.188838    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:54.272192 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.272166    2561 generic.go:356] "Generic (PLEG): container finished" podID="352e888c-4769-49f6-92be-f1b906c64832" containerID="6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69" exitCode=0
Jul 22 02:15:54.272358 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.272210    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"
Jul 22 02:15:54.272358 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.272257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" event={"ID":"352e888c-4769-49f6-92be-f1b906c64832","Type":"ContainerDied","Data":"6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69"}
Jul 22 02:15:54.272358 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.272313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" event={"ID":"352e888c-4769-49f6-92be-f1b906c64832","Type":"ContainerDied","Data":"eab10de01838fdde7c104a5225073e5bd6016f8566cd842c16cc329709cb92ea"}
Jul 22 02:15:54.272358 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.272343    2561 scope.go:117] "RemoveContainer" containerID="6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69"
Jul 22 02:15:54.280476 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.280461    2561 scope.go:117] "RemoveContainer" containerID="6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69"
Jul 22 02:15:54.280706 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:15:54.280688    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69\": container with ID starting with 6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69 not found: ID does not exist" containerID="6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69"
Jul 22 02:15:54.280775 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.280715    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69"} err="failed to get container status \"6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69\": rpc error: code = NotFound desc = could not find container \"6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69\": container with ID starting with 6d0ad3c3f677ba581f81c5aa38671dfac3eb3609ce5d2358fd134d0e881dae69 not found: ID does not exist"
Jul 22 02:15:54.372448 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.372424    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls\") pod \"352e888c-4769-49f6-92be-f1b906c64832\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") "
Jul 22 02:15:54.372589 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.372461    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352e888c-4769-49f6-92be-f1b906c64832-openshift-service-ca\") pod \"352e888c-4769-49f6-92be-f1b906c64832\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") "
Jul 22 02:15:54.372589 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.372491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrdtx\" (UniqueName: \"kubernetes.io/projected/352e888c-4769-49f6-92be-f1b906c64832-kube-api-access-qrdtx\") pod \"352e888c-4769-49f6-92be-f1b906c64832\" (UID: \"352e888c-4769-49f6-92be-f1b906c64832\") "
Jul 22 02:15:54.372805 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.372783    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352e888c-4769-49f6-92be-f1b906c64832-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "352e888c-4769-49f6-92be-f1b906c64832" (UID: "352e888c-4769-49f6-92be-f1b906c64832"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:15:54.374575 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.374548    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352e888c-4769-49f6-92be-f1b906c64832-kube-api-access-qrdtx" (OuterVolumeSpecName: "kube-api-access-qrdtx") pod "352e888c-4769-49f6-92be-f1b906c64832" (UID: "352e888c-4769-49f6-92be-f1b906c64832"). InnerVolumeSpecName "kube-api-access-qrdtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:15:54.374575 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.374554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "352e888c-4769-49f6-92be-f1b906c64832" (UID: "352e888c-4769-49f6-92be-f1b906c64832"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:15:54.473925 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.473893    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/352e888c-4769-49f6-92be-f1b906c64832-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:15:54.473925 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.473919    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/352e888c-4769-49f6-92be-f1b906c64832-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:15:54.473925 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.473929    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdtx\" (UniqueName: \"kubernetes.io/projected/352e888c-4769-49f6-92be-f1b906c64832-kube-api-access-qrdtx\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:15:54.593197 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.593167    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"]
Jul 22 02:15:54.597148 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:54.597126    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz"]
Jul 22 02:15:55.102762 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:55.102711    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="352e888c-4769-49f6-92be-f1b906c64832" path="/var/lib/kubelet/pods/352e888c-4769-49f6-92be-f1b906c64832/volumes"
Jul 22 02:15:57.172273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:57.172230    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4e4fadaa-bd4bcd6bc-jl4cz" podUID="352e888c-4769-49f6-92be-f1b906c64832" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.51:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 22 02:15:58.453137 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:58.453100    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"]
Jul 22 02:15:58.457429 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:58.457402    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4e4fadaa-j4wsj"]
Jul 22 02:15:59.102828 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:15:59.102795    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c63ddcc-546b-4df7-a6c6-c17e96c069f8" path="/var/lib/kubelet/pods/1c63ddcc-546b-4df7-a6c6-c17e96c069f8/volumes"
Jul 22 02:16:28.274893 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.274857    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"]
Jul 22 02:16:28.277187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.275191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c63ddcc-546b-4df7-a6c6-c17e96c069f8" containerName="revoke-keys"
Jul 22 02:16:28.277187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.275200    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c63ddcc-546b-4df7-a6c6-c17e96c069f8" containerName="revoke-keys"
Jul 22 02:16:28.277187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.275217    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352e888c-4769-49f6-92be-f1b906c64832" containerName="maas-api"
Jul 22 02:16:28.277187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.275225    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="352e888c-4769-49f6-92be-f1b906c64832" containerName="maas-api"
Jul 22 02:16:28.277187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.275314    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c63ddcc-546b-4df7-a6c6-c17e96c069f8" containerName="revoke-keys"
Jul 22 02:16:28.277187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.275332    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="352e888c-4769-49f6-92be-f1b906c64832" containerName="maas-api"
Jul 22 02:16:28.278143 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.278127    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.281922 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.281899    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e2f4841f-serving-cert\""
Jul 22 02:16:28.282051 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.281904    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:16:28.290874 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.290856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"]
Jul 22 02:16:28.349141 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.349111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbnb\" (UniqueName: \"kubernetes.io/projected/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-kube-api-access-plbnb\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.349342 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.349169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-openshift-service-ca\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.349342 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.349264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-maas-api-tls\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.449772 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.449739    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-maas-api-tls\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.449943 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.449785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbnb\" (UniqueName: \"kubernetes.io/projected/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-kube-api-access-plbnb\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.449997 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.449959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-openshift-service-ca\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.450597 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.450570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-openshift-service-ca\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.452171 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.452150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-maas-api-tls\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.458743 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.458701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbnb\" (UniqueName: \"kubernetes.io/projected/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-kube-api-access-plbnb\") pod \"maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.587771 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.587667    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:28.724976 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.724944    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"]
Jul 22 02:16:28.728482 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:16:28.728453    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5fa8e1_d078_4df6_be31_9fb4da2fc989.slice/crio-a3bb9db5e1b8478b17c1a5e9b5bf61ea175acdaee339a01dc2aca250c4f65d5c WatchSource:0}: Error finding container a3bb9db5e1b8478b17c1a5e9b5bf61ea175acdaee339a01dc2aca250c4f65d5c: Status 404 returned error can't find the container with id a3bb9db5e1b8478b17c1a5e9b5bf61ea175acdaee339a01dc2aca250c4f65d5c
Jul 22 02:16:28.729814 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:28.729795    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:16:29.444938 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:29.444900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg" event={"ID":"cf5fa8e1-d078-4df6-be31-9fb4da2fc989","Type":"ContainerStarted","Data":"962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c"}
Jul 22 02:16:29.445299 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:29.444963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg" event={"ID":"cf5fa8e1-d078-4df6-be31-9fb4da2fc989","Type":"ContainerStarted","Data":"a3bb9db5e1b8478b17c1a5e9b5bf61ea175acdaee339a01dc2aca250c4f65d5c"}
Jul 22 02:16:29.445299 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:29.444993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:29.464376 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:29.464330    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg" podStartSLOduration=0.901588619 podStartE2EDuration="1.464314629s" totalImagesPullingTime="562.72601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:16:29.463223435 +0000 UTC m=+1900.966081697" watchObservedRunningTime="2026-07-22 02:16:29.464314629 +0000 UTC m=+1900.967172904"
Jul 22 02:16:35.454840 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:35.454803    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:16:37.193342 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.193307    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"]
Jul 22 02:16:37.196806 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.196783    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.199683 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.199661    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2f4841f\"/\"kube-root-ca.crt\""
Jul 22 02:16:37.200958 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.200932    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2f4841f\"/\"default-dockercfg-x4w7s\""
Jul 22 02:16:37.200958 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.200941    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e2f4841f\"/\"e2e-lifecycle-model-e2f4841f-kserve-self-signed-certs\""
Jul 22 02:16:37.201102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.200965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e2f4841f\"/\"openshift-service-ca.crt\""
Jul 22 02:16:37.209736 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.209693    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"]
Jul 22 02:16:37.327172 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.327139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0548b88f-791d-4813-babb-168b13be5675-tls-certs\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.327172 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.327178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-home\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.327372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.327205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-tmp-dir\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.327372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.327234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-dshm\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.327372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.327250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-model-cache\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.327372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.327320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzdqf\" (UniqueName: \"kubernetes.io/projected/0548b88f-791d-4813-babb-168b13be5675-kube-api-access-zzdqf\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.428609 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.428578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzdqf\" (UniqueName: \"kubernetes.io/projected/0548b88f-791d-4813-babb-168b13be5675-kube-api-access-zzdqf\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.428794 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.428629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0548b88f-791d-4813-babb-168b13be5675-tls-certs\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.428794 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.428658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-home\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.428794 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.428682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-tmp-dir\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.428794 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.428714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-dshm\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.428794 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.428756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-model-cache\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.429154 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.429124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-home\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.429154 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.429150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-tmp-dir\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.429299 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.429176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-model-cache\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.430924 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.430899    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-dshm\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.431308 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.431288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0548b88f-791d-4813-babb-168b13be5675-tls-certs\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.437009 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.436977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzdqf\" (UniqueName: \"kubernetes.io/projected/0548b88f-791d-4813-babb-168b13be5675-kube-api-access-zzdqf\") pod \"e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") " pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.506900 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.506875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:37.637980 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:37.637951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"]
Jul 22 02:16:37.640600 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:16:37.640568    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0548b88f_791d_4813_babb_168b13be5675.slice/crio-bc561c244be0365fac6ff5d61a4fe76a0d552805537825f994e19bc92ccd82a9 WatchSource:0}: Error finding container bc561c244be0365fac6ff5d61a4fe76a0d552805537825f994e19bc92ccd82a9: Status 404 returned error can't find the container with id bc561c244be0365fac6ff5d61a4fe76a0d552805537825f994e19bc92ccd82a9
Jul 22 02:16:38.488230 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:38.488188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp" event={"ID":"0548b88f-791d-4813-babb-168b13be5675","Type":"ContainerStarted","Data":"071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248"}
Jul 22 02:16:38.488230 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:38.488232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp" event={"ID":"0548b88f-791d-4813-babb-168b13be5675","Type":"ContainerStarted","Data":"bc561c244be0365fac6ff5d61a4fe76a0d552805537825f994e19bc92ccd82a9"}
Jul 22 02:16:47.507505 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:47.507472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:47.507984 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:47.507523    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:47.520400 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:47.520368    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:47.539771 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:47.539724    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:16:47.540337 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:16:47.540298    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp" podStartSLOduration=10.540284867 podStartE2EDuration="10.540284867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:16:38.506509795 +0000 UTC m=+1910.009368025" watchObservedRunningTime="2026-07-22 02:16:47.540284867 +0000 UTC m=+1919.043143095"
Jul 22 02:17:12.171454 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.171420    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"]
Jul 22 02:17:12.171912 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.171685    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg" podUID="cf5fa8e1-d078-4df6-be31-9fb4da2fc989" containerName="maas-api" containerID="cri-o://962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c" gracePeriod=30
Jul 22 02:17:12.425952 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.425891    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:17:12.509881 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.509843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-maas-api-tls\") pod \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") "
Jul 22 02:17:12.510066 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.509899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-openshift-service-ca\") pod \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") "
Jul 22 02:17:12.510066 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.509935    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plbnb\" (UniqueName: \"kubernetes.io/projected/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-kube-api-access-plbnb\") pod \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\" (UID: \"cf5fa8e1-d078-4df6-be31-9fb4da2fc989\") "
Jul 22 02:17:12.510313 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.510287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf5fa8e1-d078-4df6-be31-9fb4da2fc989" (UID: "cf5fa8e1-d078-4df6-be31-9fb4da2fc989"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:17:12.511994 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.511973    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-kube-api-access-plbnb" (OuterVolumeSpecName: "kube-api-access-plbnb") pod "cf5fa8e1-d078-4df6-be31-9fb4da2fc989" (UID: "cf5fa8e1-d078-4df6-be31-9fb4da2fc989"). InnerVolumeSpecName "kube-api-access-plbnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:17:12.512071 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.512029    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf5fa8e1-d078-4df6-be31-9fb4da2fc989" (UID: "cf5fa8e1-d078-4df6-be31-9fb4da2fc989"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:17:12.610531 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.610494    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:12.610531 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.610524    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:12.610531 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.610534    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plbnb\" (UniqueName: \"kubernetes.io/projected/cf5fa8e1-d078-4df6-be31-9fb4da2fc989-kube-api-access-plbnb\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:12.648372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.648338    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf5fa8e1-d078-4df6-be31-9fb4da2fc989" containerID="962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c" exitCode=0
Jul 22 02:17:12.648505 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.648374    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg" event={"ID":"cf5fa8e1-d078-4df6-be31-9fb4da2fc989","Type":"ContainerDied","Data":"962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c"}
Jul 22 02:17:12.648505 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.648390    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"
Jul 22 02:17:12.648505 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.648434    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg" event={"ID":"cf5fa8e1-d078-4df6-be31-9fb4da2fc989","Type":"ContainerDied","Data":"a3bb9db5e1b8478b17c1a5e9b5bf61ea175acdaee339a01dc2aca250c4f65d5c"}
Jul 22 02:17:12.648505 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.648456    2561 scope.go:117] "RemoveContainer" containerID="962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c"
Jul 22 02:17:12.657228 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.657203    2561 scope.go:117] "RemoveContainer" containerID="962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c"
Jul 22 02:17:12.657468 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:17:12.657450    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c\": container with ID starting with 962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c not found: ID does not exist" containerID="962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c"
Jul 22 02:17:12.657541 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.657478    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c"} err="failed to get container status \"962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c\": rpc error: code = NotFound desc = could not find container \"962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c\": container with ID starting with 962205369ed2611e6fb7a1ac73b71e0251844ca01a8e645911e4835e66818d1c not found: ID does not exist"
Jul 22 02:17:12.670082 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.670061    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"]
Jul 22 02:17:12.674174 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:12.674153    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e2f4841f-64fdd89bcf-xmzfg"]
Jul 22 02:17:13.103057 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:13.103024    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5fa8e1-d078-4df6-be31-9fb4da2fc989" path="/var/lib/kubelet/pods/cf5fa8e1-d078-4df6-be31-9fb4da2fc989/volumes"
Jul 22 02:17:24.077719 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:24.077675    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"]
Jul 22 02:17:39.097999 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.097954    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp" podUID="0548b88f-791d-4813-babb-168b13be5675" containerName="main" containerID="cri-o://071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248" gracePeriod=45
Jul 22 02:17:39.343141 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.343116    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:17:39.433601 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-model-cache\") pod \"0548b88f-791d-4813-babb-168b13be5675\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") "
Jul 22 02:17:39.433601 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-home\") pod \"0548b88f-791d-4813-babb-168b13be5675\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") "
Jul 22 02:17:39.433601 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433585    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-dshm\") pod \"0548b88f-791d-4813-babb-168b13be5675\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") "
Jul 22 02:17:39.433890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433616    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0548b88f-791d-4813-babb-168b13be5675-tls-certs\") pod \"0548b88f-791d-4813-babb-168b13be5675\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") "
Jul 22 02:17:39.433890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzdqf\" (UniqueName: \"kubernetes.io/projected/0548b88f-791d-4813-babb-168b13be5675-kube-api-access-zzdqf\") pod \"0548b88f-791d-4813-babb-168b13be5675\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") "
Jul 22 02:17:39.433890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433744    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-tmp-dir\") pod \"0548b88f-791d-4813-babb-168b13be5675\" (UID: \"0548b88f-791d-4813-babb-168b13be5675\") "
Jul 22 02:17:39.433890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-model-cache" (OuterVolumeSpecName: "model-cache") pod "0548b88f-791d-4813-babb-168b13be5675" (UID: "0548b88f-791d-4813-babb-168b13be5675"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:17:39.433890 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.433872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-home" (OuterVolumeSpecName: "home") pod "0548b88f-791d-4813-babb-168b13be5675" (UID: "0548b88f-791d-4813-babb-168b13be5675"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:17:39.434138 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.434009    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:39.434138 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.434028    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:39.434138 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.434044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0548b88f-791d-4813-babb-168b13be5675" (UID: "0548b88f-791d-4813-babb-168b13be5675"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:17:39.435921 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.435890    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-dshm" (OuterVolumeSpecName: "dshm") pod "0548b88f-791d-4813-babb-168b13be5675" (UID: "0548b88f-791d-4813-babb-168b13be5675"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:17:39.435921 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.435893    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0548b88f-791d-4813-babb-168b13be5675-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0548b88f-791d-4813-babb-168b13be5675" (UID: "0548b88f-791d-4813-babb-168b13be5675"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:17:39.436071 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.435941    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0548b88f-791d-4813-babb-168b13be5675-kube-api-access-zzdqf" (OuterVolumeSpecName: "kube-api-access-zzdqf") pod "0548b88f-791d-4813-babb-168b13be5675" (UID: "0548b88f-791d-4813-babb-168b13be5675"). InnerVolumeSpecName "kube-api-access-zzdqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:17:39.534680 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.534653    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0548b88f-791d-4813-babb-168b13be5675-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:39.534680 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.534676    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzdqf\" (UniqueName: \"kubernetes.io/projected/0548b88f-791d-4813-babb-168b13be5675-kube-api-access-zzdqf\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:39.534861 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.534688    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:39.534861 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.534697    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0548b88f-791d-4813-babb-168b13be5675-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:17:39.778769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.778720    2561 generic.go:356] "Generic (PLEG): container finished" podID="0548b88f-791d-4813-babb-168b13be5675" containerID="071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248" exitCode=0
Jul 22 02:17:39.778937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.778782    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"
Jul 22 02:17:39.778937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.778794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp" event={"ID":"0548b88f-791d-4813-babb-168b13be5675","Type":"ContainerDied","Data":"071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248"}
Jul 22 02:17:39.778937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.778825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp" event={"ID":"0548b88f-791d-4813-babb-168b13be5675","Type":"ContainerDied","Data":"bc561c244be0365fac6ff5d61a4fe76a0d552805537825f994e19bc92ccd82a9"}
Jul 22 02:17:39.778937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.778842    2561 scope.go:117] "RemoveContainer" containerID="071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248"
Jul 22 02:17:39.790379 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.790358    2561 scope.go:117] "RemoveContainer" containerID="071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248"
Jul 22 02:17:39.790619 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:17:39.790597    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248\": container with ID starting with 071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248 not found: ID does not exist" containerID="071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248"
Jul 22 02:17:39.790690 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.790626    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248"} err="failed to get container status \"071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248\": rpc error: code = NotFound desc = could not find container \"071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248\": container with ID starting with 071e82120363e0083ccedb7fe8695f5cf34cb23ce562685d3322b30121ce6248 not found: ID does not exist"
Jul 22 02:17:39.810443 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.810421    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"]
Jul 22 02:17:39.822844 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:39.822821    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e2f4841f/e2e-lifecycle-model-e2f4841f-kserve-656558dd4c-86stp"]
Jul 22 02:17:41.102207 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:41.102175    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0548b88f-791d-4813-babb-168b13be5675" path="/var/lib/kubelet/pods/0548b88f-791d-4813-babb-168b13be5675/volumes"
Jul 22 02:17:52.043876 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:17:52.043843    2561 scope.go:117] "RemoveContainer" containerID="065e63b78ed43859edf7c733a25427de9ad026fb89ddae377bfed905d43ab4d1"
Jul 22 02:18:52.068986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:18:52.068956    2561 scope.go:117] "RemoveContainer" containerID="2f9ea1c53076b0435dfc529a0479ebeddefe03f992f2e5bbe8e53652cc92c682"
Jul 22 02:18:52.076811 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:18:52.076791    2561 scope.go:117] "RemoveContainer" containerID="86dd2861710ecc9de667ec7bd4f7972e48d2afd04eddaccd6877287606143b5e"
Jul 22 02:19:06.076370 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.076336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"]
Jul 22 02:19:06.076878 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.076860    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5fa8e1-d078-4df6-be31-9fb4da2fc989" containerName="maas-api"
Jul 22 02:19:06.076930 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.076881    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5fa8e1-d078-4df6-be31-9fb4da2fc989" containerName="maas-api"
Jul 22 02:19:06.076930 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.076918    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0548b88f-791d-4813-babb-168b13be5675" containerName="main"
Jul 22 02:19:06.076930 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.076927    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0548b88f-791d-4813-babb-168b13be5675" containerName="main"
Jul 22 02:19:06.077022 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.076999    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0548b88f-791d-4813-babb-168b13be5675" containerName="main"
Jul 22 02:19:06.077022 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.077015    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5fa8e1-d078-4df6-be31-9fb4da2fc989" containerName="maas-api"
Jul 22 02:19:06.080297 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.080267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.083291 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.083262    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:19:06.083420 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.083341    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5f059e-serving-cert\""
Jul 22 02:19:06.090913 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.090885    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"]
Jul 22 02:19:06.147337 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.147298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5x2\" (UniqueName: \"kubernetes.io/projected/337f36d6-4a33-4330-8cdb-6ec2481d4941-kube-api-access-mx5x2\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.147531 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.147363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337f36d6-4a33-4330-8cdb-6ec2481d4941-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.147531 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.147408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.248247 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.248211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5x2\" (UniqueName: \"kubernetes.io/projected/337f36d6-4a33-4330-8cdb-6ec2481d4941-kube-api-access-mx5x2\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.248455 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.248281    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337f36d6-4a33-4330-8cdb-6ec2481d4941-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.248455 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.248329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.248455 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:19:06.248446    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-serving-cert: secret "maas-api-e2e-inf-a-5f059e-serving-cert" not found
Jul 22 02:19:06.248626 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:19:06.248518    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls podName:337f36d6-4a33-4330-8cdb-6ec2481d4941 nodeName:}" failed. No retries permitted until 2026-07-22 02:19:06.7484968 +0000 UTC m=+2058.251355011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls") pod "maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" (UID: "337f36d6-4a33-4330-8cdb-6ec2481d4941") : secret "maas-api-e2e-inf-a-5f059e-serving-cert" not found
Jul 22 02:19:06.248992 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.248971    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337f36d6-4a33-4330-8cdb-6ec2481d4941-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.257223 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.257200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5x2\" (UniqueName: \"kubernetes.io/projected/337f36d6-4a33-4330-8cdb-6ec2481d4941-kube-api-access-mx5x2\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.753309 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.753272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.755630 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.755607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:06.992748 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:06.992693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:07.131089 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:07.131065    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"]
Jul 22 02:19:07.133248 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:19:07.133222    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337f36d6_4a33_4330_8cdb_6ec2481d4941.slice/crio-49d5be4499c8951df10fc54b43f57ba24767634ba7fe9474a6804921ef876314 WatchSource:0}: Error finding container 49d5be4499c8951df10fc54b43f57ba24767634ba7fe9474a6804921ef876314: Status 404 returned error can't find the container with id 49d5be4499c8951df10fc54b43f57ba24767634ba7fe9474a6804921ef876314
Jul 22 02:19:07.193566 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:07.193532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" event={"ID":"337f36d6-4a33-4330-8cdb-6ec2481d4941","Type":"ContainerStarted","Data":"49d5be4499c8951df10fc54b43f57ba24767634ba7fe9474a6804921ef876314"}
Jul 22 02:19:08.199072 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:08.199034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" event={"ID":"337f36d6-4a33-4330-8cdb-6ec2481d4941","Type":"ContainerStarted","Data":"cd9bc85b9dbd4ff9970637704e3a69c9d1515afbfc240b2dac0183eb7ed20921"}
Jul 22 02:19:08.199553 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:08.199151    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:08.222087 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:08.222038    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" podStartSLOduration=1.852379289 podStartE2EDuration="2.222022417s" totalImagesPullingTime="369.643128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:08.220024111 +0000 UTC m=+2059.722882340" watchObservedRunningTime="2026-07-22 02:19:08.222022417 +0000 UTC m=+2059.724880645"
Jul 22 02:19:14.207900 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.207815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:19:14.225633 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.225601    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"]
Jul 22 02:19:14.229472 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.229451    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.233656 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.233630    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7e7df6-serving-cert\""
Jul 22 02:19:14.239938 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.239916    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"]
Jul 22 02:19:14.319880 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.319848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.320103 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.319887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215131d-61fa-41fa-83d7-814dc09ab9de-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.320103 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.319997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fv5l\" (UniqueName: \"kubernetes.io/projected/a215131d-61fa-41fa-83d7-814dc09ab9de-kube-api-access-4fv5l\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.420536 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.420506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215131d-61fa-41fa-83d7-814dc09ab9de-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.420719 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.420562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fv5l\" (UniqueName: \"kubernetes.io/projected/a215131d-61fa-41fa-83d7-814dc09ab9de-kube-api-access-4fv5l\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.420719 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.420614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.420719 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:19:14.420708    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-serving-cert: secret "maas-api-e2e-inf-b-7e7df6-serving-cert" not found
Jul 22 02:19:14.420916 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:19:14.420800    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls podName:a215131d-61fa-41fa-83d7-814dc09ab9de nodeName:}" failed. No retries permitted until 2026-07-22 02:19:14.920784515 +0000 UTC m=+2066.423642726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls") pod "maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" (UID: "a215131d-61fa-41fa-83d7-814dc09ab9de") : secret "maas-api-e2e-inf-b-7e7df6-serving-cert" not found
Jul 22 02:19:14.421179 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.421158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215131d-61fa-41fa-83d7-814dc09ab9de-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.432068 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.432041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fv5l\" (UniqueName: \"kubernetes.io/projected/a215131d-61fa-41fa-83d7-814dc09ab9de-kube-api-access-4fv5l\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.924121 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.924090    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:14.926509 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:14.926484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls\") pod \"maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:15.142657 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:15.142621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:15.277638 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:15.277605    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"]
Jul 22 02:19:15.280607 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:19:15.280571    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda215131d_61fa_41fa_83d7_814dc09ab9de.slice/crio-ba3df6062df58fa1a809ae13ebd9838b65adf334936d2222a1d9230c711407d3 WatchSource:0}: Error finding container ba3df6062df58fa1a809ae13ebd9838b65adf334936d2222a1d9230c711407d3: Status 404 returned error can't find the container with id ba3df6062df58fa1a809ae13ebd9838b65adf334936d2222a1d9230c711407d3
Jul 22 02:19:16.240077 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:16.240044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" event={"ID":"a215131d-61fa-41fa-83d7-814dc09ab9de","Type":"ContainerStarted","Data":"3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6"}
Jul 22 02:19:16.240288 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:16.240094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" event={"ID":"a215131d-61fa-41fa-83d7-814dc09ab9de","Type":"ContainerStarted","Data":"ba3df6062df58fa1a809ae13ebd9838b65adf334936d2222a1d9230c711407d3"}
Jul 22 02:19:16.240288 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:16.240138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:16.262463 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:16.262400    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" podStartSLOduration=1.874192493 podStartE2EDuration="2.262382049s" totalImagesPullingTime="388.189556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:16.260635828 +0000 UTC m=+2067.763494093" watchObservedRunningTime="2026-07-22 02:19:16.262382049 +0000 UTC m=+2067.765240282"
Jul 22 02:19:22.251210 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:22.251176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:19:27.490700 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.490664    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"]
Jul 22 02:19:27.494027 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.494007    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.496442 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.496422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f059e\"/\"default-dockercfg-5m259\""
Jul 22 02:19:27.496521 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.496423    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f059e\"/\"kube-root-ca.crt\""
Jul 22 02:19:27.497516 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.497501    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5f059e\"/\"test-model-5f059e-kserve-self-signed-certs\""
Jul 22 02:19:27.497516 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.497510    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5f059e\"/\"openshift-service-ca.crt\""
Jul 22 02:19:27.504912 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.504888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"]
Jul 22 02:19:27.525266 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.525238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-home\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.525402 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.525277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-dshm\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.525402 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.525352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-model-cache\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.525526 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.525446    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4a4585-6443-47f7-9451-061102330742-tls-certs\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.525526 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.525492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-tmp-dir\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.525526 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.525520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbznw\" (UniqueName: \"kubernetes.io/projected/cf4a4585-6443-47f7-9451-061102330742-kube-api-access-wbznw\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.626709 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.626674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-home\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.626709 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.626707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-dshm\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.626917 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.626899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-model-cache\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.626971 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.626959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4a4585-6443-47f7-9451-061102330742-tls-certs\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.627027 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.626989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-tmp-dir\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.627027 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.627009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbznw\" (UniqueName: \"kubernetes.io/projected/cf4a4585-6443-47f7-9451-061102330742-kube-api-access-wbznw\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.627139 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.627073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-home\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.627264 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.627241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-model-cache\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.627368 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.627346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-tmp-dir\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.629011 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.628985    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-dshm\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.629276 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.629258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4a4585-6443-47f7-9451-061102330742-tls-certs\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.635141 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.635119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbznw\" (UniqueName: \"kubernetes.io/projected/cf4a4585-6443-47f7-9451-061102330742-kube-api-access-wbznw\") pod \"test-model-5f059e-kserve-77854bd669-r2ff8\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") " pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.803697 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.803657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:27.937094 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:27.936198    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"]
Jul 22 02:19:27.939954 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:19:27.939916    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf4a4585_6443_47f7_9451_061102330742.slice/crio-e2a8120871d8ba0eec16f915fab894453ce97fd9fd22fe995bbf2a46d2467f93 WatchSource:0}: Error finding container e2a8120871d8ba0eec16f915fab894453ce97fd9fd22fe995bbf2a46d2467f93: Status 404 returned error can't find the container with id e2a8120871d8ba0eec16f915fab894453ce97fd9fd22fe995bbf2a46d2467f93
Jul 22 02:19:28.298519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:28.298483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8" event={"ID":"cf4a4585-6443-47f7-9451-061102330742","Type":"ContainerStarted","Data":"b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee"}
Jul 22 02:19:28.298693 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:28.298529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8" event={"ID":"cf4a4585-6443-47f7-9451-061102330742","Type":"ContainerStarted","Data":"e2a8120871d8ba0eec16f915fab894453ce97fd9fd22fe995bbf2a46d2467f93"}
Jul 22 02:19:28.318304 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:28.318246    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8" podStartSLOduration=1.318229702 podStartE2EDuration="1.318229702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:19:28.316511709 +0000 UTC m=+2079.819369938" watchObservedRunningTime="2026-07-22 02:19:28.318229702 +0000 UTC m=+2079.821087931"
Jul 22 02:19:37.803937 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:37.803896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:37.804375 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:37.803951    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:37.816596 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:37.816567    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:38.363105 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:38.363078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:19:49.121620 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:49.121583    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:19:49.122480 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:49.122452    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:19:52.096245 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:19:52.096213    2561 scope.go:117] "RemoveContainer" containerID="bc2a3facc5edb29df6a0d0d62bcc4261d67c9a89b236e22701896d78f238b2f7"
Jul 22 02:20:11.319019 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.318984    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"]
Jul 22 02:20:11.322492 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.322473    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.325160 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.325130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7e7df6\"/\"openshift-service-ca.crt\""
Jul 22 02:20:11.325160 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.325148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7e7df6\"/\"test-model-7e7df6-kserve-self-signed-certs\""
Jul 22 02:20:11.326098 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.326078    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7e7df6\"/\"kube-root-ca.crt\""
Jul 22 02:20:11.326200 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.326186    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7e7df6\"/\"default-dockercfg-rknsx\""
Jul 22 02:20:11.339066 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.339044    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"]
Jul 22 02:20:11.398413 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.398386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-tmp-dir\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.398413 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.398416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36919759-0e89-48fc-b686-9082fd571a6b-tls-certs\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.398612 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.398451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-dshm\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.398612 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.398527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxls\" (UniqueName: \"kubernetes.io/projected/36919759-0e89-48fc-b686-9082fd571a6b-kube-api-access-drxls\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.398612 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.398595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-home\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.398717 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.398662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-model-cache\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.499547 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-home\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.499765 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-model-cache\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.499765 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-tmp-dir\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.499765 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499630    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36919759-0e89-48fc-b686-9082fd571a6b-tls-certs\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.499765 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499679    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-dshm\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.499765 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499707    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxls\" (UniqueName: \"kubernetes.io/projected/36919759-0e89-48fc-b686-9082fd571a6b-kube-api-access-drxls\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.500058 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.499995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-home\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.500058 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.500032    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-tmp-dir\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.500153 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.500047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-model-cache\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.502028 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.502005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-dshm\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.502202 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.502182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36919759-0e89-48fc-b686-9082fd571a6b-tls-certs\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.509241 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.509219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxls\" (UniqueName: \"kubernetes.io/projected/36919759-0e89-48fc-b686-9082fd571a6b-kube-api-access-drxls\") pod \"test-model-7e7df6-kserve-d6b65d65f-p9xxv\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") " pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.632486 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.632404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:11.766341 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:11.766316    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"]
Jul 22 02:20:11.768474 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:20:11.768446    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36919759_0e89_48fc_b686_9082fd571a6b.slice/crio-54b8d8d0b5c7c821c3fb43176db3762a36cc95a4df46d9fb8e9851d6c44c29ce WatchSource:0}: Error finding container 54b8d8d0b5c7c821c3fb43176db3762a36cc95a4df46d9fb8e9851d6c44c29ce: Status 404 returned error can't find the container with id 54b8d8d0b5c7c821c3fb43176db3762a36cc95a4df46d9fb8e9851d6c44c29ce
Jul 22 02:20:12.508482 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:12.508442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv" event={"ID":"36919759-0e89-48fc-b686-9082fd571a6b","Type":"ContainerStarted","Data":"d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac"}
Jul 22 02:20:12.508959 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:12.508496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv" event={"ID":"36919759-0e89-48fc-b686-9082fd571a6b","Type":"ContainerStarted","Data":"54b8d8d0b5c7c821c3fb43176db3762a36cc95a4df46d9fb8e9851d6c44c29ce"}
Jul 22 02:20:12.526258 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:12.526216    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv" podStartSLOduration=1.526202179 podStartE2EDuration="1.526202179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:20:12.525434728 +0000 UTC m=+2124.028292958" watchObservedRunningTime="2026-07-22 02:20:12.526202179 +0000 UTC m=+2124.029060408"
Jul 22 02:20:21.632901 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:21.632866    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:21.633377 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:21.632921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:21.650093 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:21.650064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:22.565433 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:22.565407    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:20:53.836106 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:53.836021    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"]
Jul 22 02:20:54.764670 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:20:54.764638    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"]
Jul 22 02:21:05.604687 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.604651    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"]
Jul 22 02:21:05.605394 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.604963    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" podUID="337f36d6-4a33-4330-8cdb-6ec2481d4941" containerName="maas-api" containerID="cri-o://cd9bc85b9dbd4ff9970637704e3a69c9d1515afbfc240b2dac0183eb7ed20921" gracePeriod=30
Jul 22 02:21:05.767142 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.767106    2561 generic.go:356] "Generic (PLEG): container finished" podID="337f36d6-4a33-4330-8cdb-6ec2481d4941" containerID="cd9bc85b9dbd4ff9970637704e3a69c9d1515afbfc240b2dac0183eb7ed20921" exitCode=0
Jul 22 02:21:05.767302 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.767172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" event={"ID":"337f36d6-4a33-4330-8cdb-6ec2481d4941","Type":"ContainerDied","Data":"cd9bc85b9dbd4ff9970637704e3a69c9d1515afbfc240b2dac0183eb7ed20921"}
Jul 22 02:21:05.859651 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.859597    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:21:05.968085 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.968057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx5x2\" (UniqueName: \"kubernetes.io/projected/337f36d6-4a33-4330-8cdb-6ec2481d4941-kube-api-access-mx5x2\") pod \"337f36d6-4a33-4330-8cdb-6ec2481d4941\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") "
Jul 22 02:21:05.968232 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.968098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls\") pod \"337f36d6-4a33-4330-8cdb-6ec2481d4941\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") "
Jul 22 02:21:05.968232 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.968140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337f36d6-4a33-4330-8cdb-6ec2481d4941-openshift-service-ca\") pod \"337f36d6-4a33-4330-8cdb-6ec2481d4941\" (UID: \"337f36d6-4a33-4330-8cdb-6ec2481d4941\") "
Jul 22 02:21:05.968593 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.968564    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337f36d6-4a33-4330-8cdb-6ec2481d4941-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "337f36d6-4a33-4330-8cdb-6ec2481d4941" (UID: "337f36d6-4a33-4330-8cdb-6ec2481d4941"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:21:05.970142 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.970119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "337f36d6-4a33-4330-8cdb-6ec2481d4941" (UID: "337f36d6-4a33-4330-8cdb-6ec2481d4941"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:21:05.970241 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:05.970121    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337f36d6-4a33-4330-8cdb-6ec2481d4941-kube-api-access-mx5x2" (OuterVolumeSpecName: "kube-api-access-mx5x2") pod "337f36d6-4a33-4330-8cdb-6ec2481d4941" (UID: "337f36d6-4a33-4330-8cdb-6ec2481d4941"). InnerVolumeSpecName "kube-api-access-mx5x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:21:06.069337 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.069307    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5x2\" (UniqueName: \"kubernetes.io/projected/337f36d6-4a33-4330-8cdb-6ec2481d4941-kube-api-access-mx5x2\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:06.069470 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.069333    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337f36d6-4a33-4330-8cdb-6ec2481d4941-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:06.069470 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.069353    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337f36d6-4a33-4330-8cdb-6ec2481d4941-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:06.773807 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.773767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb" event={"ID":"337f36d6-4a33-4330-8cdb-6ec2481d4941","Type":"ContainerDied","Data":"49d5be4499c8951df10fc54b43f57ba24767634ba7fe9474a6804921ef876314"}
Jul 22 02:21:06.773807 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.773803    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"
Jul 22 02:21:06.774272 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.773838    2561 scope.go:117] "RemoveContainer" containerID="cd9bc85b9dbd4ff9970637704e3a69c9d1515afbfc240b2dac0183eb7ed20921"
Jul 22 02:21:06.798263 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.798231    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"]
Jul 22 02:21:06.801764 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:06.801742    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f059e-74fc7f84d5-rg6qb"]
Jul 22 02:21:07.102928 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:07.102847    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="337f36d6-4a33-4330-8cdb-6ec2481d4941" path="/var/lib/kubelet/pods/337f36d6-4a33-4330-8cdb-6ec2481d4941/volumes"
Jul 22 02:21:08.856617 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:08.856560    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8" podUID="cf4a4585-6443-47f7-9451-061102330742" containerName="main" containerID="cri-o://b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee" gracePeriod=45
Jul 22 02:21:09.110449 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.110389    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:21:09.195121 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195089    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-tmp-dir\") pod \"cf4a4585-6443-47f7-9451-061102330742\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") "
Jul 22 02:21:09.195286 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195132    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-home\") pod \"cf4a4585-6443-47f7-9451-061102330742\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") "
Jul 22 02:21:09.195286 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195163    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-dshm\") pod \"cf4a4585-6443-47f7-9451-061102330742\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") "
Jul 22 02:21:09.195286 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4a4585-6443-47f7-9451-061102330742-tls-certs\") pod \"cf4a4585-6443-47f7-9451-061102330742\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") "
Jul 22 02:21:09.195286 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbznw\" (UniqueName: \"kubernetes.io/projected/cf4a4585-6443-47f7-9451-061102330742-kube-api-access-wbznw\") pod \"cf4a4585-6443-47f7-9451-061102330742\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") "
Jul 22 02:21:09.195286 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195279    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-model-cache\") pod \"cf4a4585-6443-47f7-9451-061102330742\" (UID: \"cf4a4585-6443-47f7-9451-061102330742\") "
Jul 22 02:21:09.195502 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf4a4585-6443-47f7-9451-061102330742" (UID: "cf4a4585-6443-47f7-9451-061102330742"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:09.195502 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195405    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-home" (OuterVolumeSpecName: "home") pod "cf4a4585-6443-47f7-9451-061102330742" (UID: "cf4a4585-6443-47f7-9451-061102330742"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:09.195589 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195574    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf4a4585-6443-47f7-9451-061102330742" (UID: "cf4a4585-6443-47f7-9451-061102330742"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:09.195654 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195602    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:09.195654 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.195631    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:09.197498 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.197470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-dshm" (OuterVolumeSpecName: "dshm") pod "cf4a4585-6443-47f7-9451-061102330742" (UID: "cf4a4585-6443-47f7-9451-061102330742"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:09.197606 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.197522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4a4585-6443-47f7-9451-061102330742-kube-api-access-wbznw" (OuterVolumeSpecName: "kube-api-access-wbznw") pod "cf4a4585-6443-47f7-9451-061102330742" (UID: "cf4a4585-6443-47f7-9451-061102330742"). InnerVolumeSpecName "kube-api-access-wbznw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:21:09.197606 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.197570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4a4585-6443-47f7-9451-061102330742-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf4a4585-6443-47f7-9451-061102330742" (UID: "cf4a4585-6443-47f7-9451-061102330742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:21:09.296351 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.296320    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:09.296351 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.296345    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4a4585-6443-47f7-9451-061102330742-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:09.296351 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.296355    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbznw\" (UniqueName: \"kubernetes.io/projected/cf4a4585-6443-47f7-9451-061102330742-kube-api-access-wbznw\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:09.296541 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.296365    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4a4585-6443-47f7-9451-061102330742-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:09.785267 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.785224    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv" podUID="36919759-0e89-48fc-b686-9082fd571a6b" containerName="main" containerID="cri-o://d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac" gracePeriod=45
Jul 22 02:21:09.791141 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.791106    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf4a4585-6443-47f7-9451-061102330742" containerID="b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee" exitCode=0
Jul 22 02:21:09.791246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.791154    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"
Jul 22 02:21:09.791246 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.791149    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8" event={"ID":"cf4a4585-6443-47f7-9451-061102330742","Type":"ContainerDied","Data":"b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee"}
Jul 22 02:21:09.791333 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.791263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8" event={"ID":"cf4a4585-6443-47f7-9451-061102330742","Type":"ContainerDied","Data":"e2a8120871d8ba0eec16f915fab894453ce97fd9fd22fe995bbf2a46d2467f93"}
Jul 22 02:21:09.791333 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.791291    2561 scope.go:117] "RemoveContainer" containerID="b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee"
Jul 22 02:21:09.856096 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.856068    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"]
Jul 22 02:21:09.861762 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.861739    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5f059e/test-model-5f059e-kserve-77854bd669-r2ff8"]
Jul 22 02:21:09.895490 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.895438    2561 scope.go:117] "RemoveContainer" containerID="b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee"
Jul 22 02:21:09.895829 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:21:09.895802    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee\": container with ID starting with b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee not found: ID does not exist" containerID="b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee"
Jul 22 02:21:09.895883 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:09.895843    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee"} err="failed to get container status \"b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee\": rpc error: code = NotFound desc = could not find container \"b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee\": container with ID starting with b526a9f6c111f1609f3e0bd0996e6bd1e1af27e10724ce8201323e21c41fc2ee not found: ID does not exist"
Jul 22 02:21:10.042612 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.042539    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:21:10.202682 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.202651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-home\") pod \"36919759-0e89-48fc-b686-9082fd571a6b\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") "
Jul 22 02:21:10.202817 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.202707    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-model-cache\") pod \"36919759-0e89-48fc-b686-9082fd571a6b\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") "
Jul 22 02:21:10.202817 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.202752    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-tmp-dir\") pod \"36919759-0e89-48fc-b686-9082fd571a6b\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") "
Jul 22 02:21:10.202817 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.202768    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drxls\" (UniqueName: \"kubernetes.io/projected/36919759-0e89-48fc-b686-9082fd571a6b-kube-api-access-drxls\") pod \"36919759-0e89-48fc-b686-9082fd571a6b\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") "
Jul 22 02:21:10.202817 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.202799    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36919759-0e89-48fc-b686-9082fd571a6b-tls-certs\") pod \"36919759-0e89-48fc-b686-9082fd571a6b\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") "
Jul 22 02:21:10.202995 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.202827    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-dshm\") pod \"36919759-0e89-48fc-b686-9082fd571a6b\" (UID: \"36919759-0e89-48fc-b686-9082fd571a6b\") "
Jul 22 02:21:10.203716 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.203263    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "36919759-0e89-48fc-b686-9082fd571a6b" (UID: "36919759-0e89-48fc-b686-9082fd571a6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:10.203716 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.203286    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36919759-0e89-48fc-b686-9082fd571a6b" (UID: "36919759-0e89-48fc-b686-9082fd571a6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:10.203716 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.203497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-home" (OuterVolumeSpecName: "home") pod "36919759-0e89-48fc-b686-9082fd571a6b" (UID: "36919759-0e89-48fc-b686-9082fd571a6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:10.203716 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.203643    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:10.203716 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.203684    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:10.203716 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.203701    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:10.208779 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.207130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-dshm" (OuterVolumeSpecName: "dshm") pod "36919759-0e89-48fc-b686-9082fd571a6b" (UID: "36919759-0e89-48fc-b686-9082fd571a6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:21:10.208779 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.207181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36919759-0e89-48fc-b686-9082fd571a6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36919759-0e89-48fc-b686-9082fd571a6b" (UID: "36919759-0e89-48fc-b686-9082fd571a6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:21:10.210673 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.210583    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36919759-0e89-48fc-b686-9082fd571a6b-kube-api-access-drxls" (OuterVolumeSpecName: "kube-api-access-drxls") pod "36919759-0e89-48fc-b686-9082fd571a6b" (UID: "36919759-0e89-48fc-b686-9082fd571a6b"). InnerVolumeSpecName "kube-api-access-drxls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:21:10.304216 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.304121    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxls\" (UniqueName: \"kubernetes.io/projected/36919759-0e89-48fc-b686-9082fd571a6b-kube-api-access-drxls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:10.304216 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.304157    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36919759-0e89-48fc-b686-9082fd571a6b-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:10.304216 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.304168    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36919759-0e89-48fc-b686-9082fd571a6b-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:10.797402 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.797367    2561 generic.go:356] "Generic (PLEG): container finished" podID="36919759-0e89-48fc-b686-9082fd571a6b" containerID="d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac" exitCode=0
Jul 22 02:21:10.797587 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.797421    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"
Jul 22 02:21:10.797587 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.797460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv" event={"ID":"36919759-0e89-48fc-b686-9082fd571a6b","Type":"ContainerDied","Data":"d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac"}
Jul 22 02:21:10.797587 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.797507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv" event={"ID":"36919759-0e89-48fc-b686-9082fd571a6b","Type":"ContainerDied","Data":"54b8d8d0b5c7c821c3fb43176db3762a36cc95a4df46d9fb8e9851d6c44c29ce"}
Jul 22 02:21:10.797587 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.797527    2561 scope.go:117] "RemoveContainer" containerID="d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac"
Jul 22 02:21:10.806661 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.806643    2561 scope.go:117] "RemoveContainer" containerID="d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac"
Jul 22 02:21:10.806969 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:21:10.806947    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac\": container with ID starting with d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac not found: ID does not exist" containerID="d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac"
Jul 22 02:21:10.807038 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.806975    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac"} err="failed to get container status \"d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac\": rpc error: code = NotFound desc = could not find container \"d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac\": container with ID starting with d913c0635be2a749f3ef259e6434d6043f684a703820b10b02b1e828536cbbac not found: ID does not exist"
Jul 22 02:21:10.831222 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.831195    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"]
Jul 22 02:21:10.832640 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:10.832620    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7e7df6/test-model-7e7df6-kserve-d6b65d65f-p9xxv"]
Jul 22 02:21:11.101939 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:11.101859    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36919759-0e89-48fc-b686-9082fd571a6b" path="/var/lib/kubelet/pods/36919759-0e89-48fc-b686-9082fd571a6b/volumes"
Jul 22 02:21:11.102305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:11.102244    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf4a4585-6443-47f7-9451-061102330742" path="/var/lib/kubelet/pods/cf4a4585-6443-47f7-9451-061102330742/volumes"
Jul 22 02:21:17.101784 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.101755    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"]
Jul 22 02:21:17.102236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102095    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="337f36d6-4a33-4330-8cdb-6ec2481d4941" containerName="maas-api"
Jul 22 02:21:17.102236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102110    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="337f36d6-4a33-4330-8cdb-6ec2481d4941" containerName="maas-api"
Jul 22 02:21:17.102236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102129    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4a4585-6443-47f7-9451-061102330742" containerName="main"
Jul 22 02:21:17.102236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102137    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4a4585-6443-47f7-9451-061102330742" containerName="main"
Jul 22 02:21:17.102236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102151    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36919759-0e89-48fc-b686-9082fd571a6b" containerName="main"
Jul 22 02:21:17.102236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102158    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="36919759-0e89-48fc-b686-9082fd571a6b" containerName="main"
Jul 22 02:21:17.102521 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102263    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4a4585-6443-47f7-9451-061102330742" containerName="main"
Jul 22 02:21:17.102521 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102277    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="337f36d6-4a33-4330-8cdb-6ec2481d4941" containerName="maas-api"
Jul 22 02:21:17.102521 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.102285    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="36919759-0e89-48fc-b686-9082fd571a6b" containerName="main"
Jul 22 02:21:17.106810 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.106793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:17.109984 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.109963    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:21:17.118143 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.118125    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"]
Jul 22 02:21:17.164941 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.164919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ab93b3-39d2-4b81-9215-1302d0f45999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w\" (UID: \"72ab93b3-39d2-4b81-9215-1302d0f45999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:17.265804 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.265774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ab93b3-39d2-4b81-9215-1302d0f45999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w\" (UID: \"72ab93b3-39d2-4b81-9215-1302d0f45999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:17.266476 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.266457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ab93b3-39d2-4b81-9215-1302d0f45999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w\" (UID: \"72ab93b3-39d2-4b81-9215-1302d0f45999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:17.416805 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.416711    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:17.551805 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.551712    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"]
Jul 22 02:21:17.554075 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:21:17.554046    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ab93b3_39d2_4b81_9215_1302d0f45999.slice/crio-fa05188acbf789cba3916bad121108e925f051f1027a2d27efe6ab50d2900a10 WatchSource:0}: Error finding container fa05188acbf789cba3916bad121108e925f051f1027a2d27efe6ab50d2900a10: Status 404 returned error can't find the container with id fa05188acbf789cba3916bad121108e925f051f1027a2d27efe6ab50d2900a10
Jul 22 02:21:17.831423 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.831391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w" event={"ID":"72ab93b3-39d2-4b81-9215-1302d0f45999","Type":"ContainerStarted","Data":"2a88bcdc6f87b36ddac621c072441963a5e97480e54c16ded1d92fd8388f7e7c"}
Jul 22 02:21:17.831591 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.831445    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w" event={"ID":"72ab93b3-39d2-4b81-9215-1302d0f45999","Type":"ContainerStarted","Data":"fa05188acbf789cba3916bad121108e925f051f1027a2d27efe6ab50d2900a10"}
Jul 22 02:21:17.862159 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:17.862115    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w" podStartSLOduration=0.862100094 podStartE2EDuration="862.100094ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:17.860554053 +0000 UTC m=+2189.363412280" watchObservedRunningTime="2026-07-22 02:21:17.862100094 +0000 UTC m=+2189.364958348"
Jul 22 02:21:18.837573 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:18.837531    2561 generic.go:356] "Generic (PLEG): container finished" podID="72ab93b3-39d2-4b81-9215-1302d0f45999" containerID="2a88bcdc6f87b36ddac621c072441963a5e97480e54c16ded1d92fd8388f7e7c" exitCode=0
Jul 22 02:21:18.837986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:18.837598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w" event={"ID":"72ab93b3-39d2-4b81-9215-1302d0f45999","Type":"ContainerDied","Data":"2a88bcdc6f87b36ddac621c072441963a5e97480e54c16ded1d92fd8388f7e7c"}
Jul 22 02:21:19.976705 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:19.976684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:19.986477 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:19.986456    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ab93b3-39d2-4b81-9215-1302d0f45999-maas-api-service-ca\") pod \"72ab93b3-39d2-4b81-9215-1302d0f45999\" (UID: \"72ab93b3-39d2-4b81-9215-1302d0f45999\") "
Jul 22 02:21:19.986822 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:19.986799    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ab93b3-39d2-4b81-9215-1302d0f45999-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72ab93b3-39d2-4b81-9215-1302d0f45999" (UID: "72ab93b3-39d2-4b81-9215-1302d0f45999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:21:20.087966 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:20.087932    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ab93b3-39d2-4b81-9215-1302d0f45999-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:20.850385 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:20.850353    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"
Jul 22 02:21:20.850555 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:20.850383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w" event={"ID":"72ab93b3-39d2-4b81-9215-1302d0f45999","Type":"ContainerDied","Data":"fa05188acbf789cba3916bad121108e925f051f1027a2d27efe6ab50d2900a10"}
Jul 22 02:21:20.850555 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:20.850429    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa05188acbf789cba3916bad121108e925f051f1027a2d27efe6ab50d2900a10"
Jul 22 02:21:27.572960 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.572927    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"]
Jul 22 02:21:27.573379 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.573155    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" podUID="a215131d-61fa-41fa-83d7-814dc09ab9de" containerName="maas-api" containerID="cri-o://3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6" gracePeriod=30
Jul 22 02:21:27.817891 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.817866    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:21:27.847353 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.847287    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls\") pod \"a215131d-61fa-41fa-83d7-814dc09ab9de\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") "
Jul 22 02:21:27.847353 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.847336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215131d-61fa-41fa-83d7-814dc09ab9de-openshift-service-ca\") pod \"a215131d-61fa-41fa-83d7-814dc09ab9de\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") "
Jul 22 02:21:27.847501 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.847381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fv5l\" (UniqueName: \"kubernetes.io/projected/a215131d-61fa-41fa-83d7-814dc09ab9de-kube-api-access-4fv5l\") pod \"a215131d-61fa-41fa-83d7-814dc09ab9de\" (UID: \"a215131d-61fa-41fa-83d7-814dc09ab9de\") "
Jul 22 02:21:27.847795 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.847770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a215131d-61fa-41fa-83d7-814dc09ab9de-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a215131d-61fa-41fa-83d7-814dc09ab9de" (UID: "a215131d-61fa-41fa-83d7-814dc09ab9de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:21:27.849319 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.849239    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a215131d-61fa-41fa-83d7-814dc09ab9de" (UID: "a215131d-61fa-41fa-83d7-814dc09ab9de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:21:27.849430 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.849333    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a215131d-61fa-41fa-83d7-814dc09ab9de-kube-api-access-4fv5l" (OuterVolumeSpecName: "kube-api-access-4fv5l") pod "a215131d-61fa-41fa-83d7-814dc09ab9de" (UID: "a215131d-61fa-41fa-83d7-814dc09ab9de"). InnerVolumeSpecName "kube-api-access-4fv5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:21:27.884424 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.884395    2561 generic.go:356] "Generic (PLEG): container finished" podID="a215131d-61fa-41fa-83d7-814dc09ab9de" containerID="3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6" exitCode=0
Jul 22 02:21:27.884523 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.884432    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"
Jul 22 02:21:27.884523 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.884474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" event={"ID":"a215131d-61fa-41fa-83d7-814dc09ab9de","Type":"ContainerDied","Data":"3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6"}
Jul 22 02:21:27.884523 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.884521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs" event={"ID":"a215131d-61fa-41fa-83d7-814dc09ab9de","Type":"ContainerDied","Data":"ba3df6062df58fa1a809ae13ebd9838b65adf334936d2222a1d9230c711407d3"}
Jul 22 02:21:27.884632 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.884542    2561 scope.go:117] "RemoveContainer" containerID="3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6"
Jul 22 02:21:27.893630 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.893612    2561 scope.go:117] "RemoveContainer" containerID="3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6"
Jul 22 02:21:27.893897 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:21:27.893880    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6\": container with ID starting with 3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6 not found: ID does not exist" containerID="3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6"
Jul 22 02:21:27.893960 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.893905    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6"} err="failed to get container status \"3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6\": rpc error: code = NotFound desc = could not find container \"3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6\": container with ID starting with 3878c69849836c9293e0244bad8028487096a3e89dc0356892bb6d57570eb6e6 not found: ID does not exist"
Jul 22 02:21:27.905256 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.905235    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"]
Jul 22 02:21:27.909488 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.909465    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7e7df6-bb9895bc9-c98gs"]
Jul 22 02:21:27.948880 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.948855    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a215131d-61fa-41fa-83d7-814dc09ab9de-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:27.948880 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.948878    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fv5l\" (UniqueName: \"kubernetes.io/projected/a215131d-61fa-41fa-83d7-814dc09ab9de-kube-api-access-4fv5l\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:27.948880 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:27.948892    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a215131d-61fa-41fa-83d7-814dc09ab9de-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:21:29.102173 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:29.102140    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a215131d-61fa-41fa-83d7-814dc09ab9de" path="/var/lib/kubelet/pods/a215131d-61fa-41fa-83d7-814dc09ab9de/volumes"
Jul 22 02:21:32.206213 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:32.206178    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"]
Jul 22 02:21:32.207759 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:32.207719    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7e7df6-nz47w"]
Jul 22 02:21:33.101817 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:33.101787    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ab93b3-39d2-4b81-9215-1302d0f45999" path="/var/lib/kubelet/pods/72ab93b3-39d2-4b81-9215-1302d0f45999/volumes"
Jul 22 02:21:43.104848 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.104819    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"]
Jul 22 02:21:43.105285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.105112    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a215131d-61fa-41fa-83d7-814dc09ab9de" containerName="maas-api"
Jul 22 02:21:43.105285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.105122    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a215131d-61fa-41fa-83d7-814dc09ab9de" containerName="maas-api"
Jul 22 02:21:43.105285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.105138    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ab93b3-39d2-4b81-9215-1302d0f45999" containerName="revoke-keys"
Jul 22 02:21:43.105285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.105144    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ab93b3-39d2-4b81-9215-1302d0f45999" containerName="revoke-keys"
Jul 22 02:21:43.105285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.105200    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a215131d-61fa-41fa-83d7-814dc09ab9de" containerName="maas-api"
Jul 22 02:21:43.105285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.105210    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ab93b3-39d2-4b81-9215-1302d0f45999" containerName="revoke-keys"
Jul 22 02:21:43.108334 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.108316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.113343 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.113319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:21:43.113343 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.113336    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-633172-serving-cert\""
Jul 22 02:21:43.120675 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.120642    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"]
Jul 22 02:21:43.173199 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.173173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ea217-ec2e-41f3-9062-632cc7e9a17e-maas-api-tls\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.173342 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.173216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn96j\" (UniqueName: \"kubernetes.io/projected/e27ea217-ec2e-41f3-9062-632cc7e9a17e-kube-api-access-fn96j\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.173342 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.173305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ea217-ec2e-41f3-9062-632cc7e9a17e-openshift-service-ca\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.274675 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.274645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ea217-ec2e-41f3-9062-632cc7e9a17e-maas-api-tls\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.274858 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.274698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn96j\" (UniqueName: \"kubernetes.io/projected/e27ea217-ec2e-41f3-9062-632cc7e9a17e-kube-api-access-fn96j\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.274858 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.274771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ea217-ec2e-41f3-9062-632cc7e9a17e-openshift-service-ca\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.275417 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.275396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ea217-ec2e-41f3-9062-632cc7e9a17e-openshift-service-ca\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.277123 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.277094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ea217-ec2e-41f3-9062-632cc7e9a17e-maas-api-tls\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.284989 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.284969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn96j\" (UniqueName: \"kubernetes.io/projected/e27ea217-ec2e-41f3-9062-632cc7e9a17e-kube-api-access-fn96j\") pod \"maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.418750 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.418667    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:43.553748 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.553696    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"]
Jul 22 02:21:43.556158 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:21:43.556126    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27ea217_ec2e_41f3_9062_632cc7e9a17e.slice/crio-8b4039263dca23b4272b054adee18cb37fb0c67808061f62d19e1ed3d0adfc4b WatchSource:0}: Error finding container 8b4039263dca23b4272b054adee18cb37fb0c67808061f62d19e1ed3d0adfc4b: Status 404 returned error can't find the container with id 8b4039263dca23b4272b054adee18cb37fb0c67808061f62d19e1ed3d0adfc4b
Jul 22 02:21:43.557515 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.557495    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:21:43.964655 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:43.964621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b" event={"ID":"e27ea217-ec2e-41f3-9062-632cc7e9a17e","Type":"ContainerStarted","Data":"8b4039263dca23b4272b054adee18cb37fb0c67808061f62d19e1ed3d0adfc4b"}
Jul 22 02:21:44.970837 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:44.970801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b" event={"ID":"e27ea217-ec2e-41f3-9062-632cc7e9a17e","Type":"ContainerStarted","Data":"f67e17a8325fc39a3cdf1d6920012427a2c3a05c35619309bf9e2ad888584554"}
Jul 22 02:21:44.971253 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:44.970927    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:44.991914 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:44.991868    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b" podStartSLOduration=1.42122667 podStartE2EDuration="1.991850823s" totalImagesPullingTime="570.624153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:44.989655819 +0000 UTC m=+2216.492514049" watchObservedRunningTime="2026-07-22 02:21:44.991850823 +0000 UTC m=+2216.494709079"
Jul 22 02:21:50.980505 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:50.980471    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:21:51.283972 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.283937    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"]
Jul 22 02:21:51.287340 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.287325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.290938 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.290913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dad16f-serving-cert\""
Jul 22 02:21:51.316682 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.316659    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"]
Jul 22 02:21:51.340481 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.340450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.340622 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.340505    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d0eb9-4dbb-4166-838d-1a90057b887d-openshift-service-ca\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.340684 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.340653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn887\" (UniqueName: \"kubernetes.io/projected/8a8d0eb9-4dbb-4166-838d-1a90057b887d-kube-api-access-fn887\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.441836 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.441806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn887\" (UniqueName: \"kubernetes.io/projected/8a8d0eb9-4dbb-4166-838d-1a90057b887d-kube-api-access-fn887\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.441836 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.441839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.442088 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.441862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d0eb9-4dbb-4166-838d-1a90057b887d-openshift-service-ca\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.442088 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:21:51.441967    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-serving-cert: secret "maas-api-e2e-api-b-dad16f-serving-cert" not found
Jul 22 02:21:51.442088 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:21:51.442033    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls podName:8a8d0eb9-4dbb-4166-838d-1a90057b887d nodeName:}" failed. No retries permitted until 2026-07-22 02:21:51.942015226 +0000 UTC m=+2223.444873438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls") pod "maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" (UID: "8a8d0eb9-4dbb-4166-838d-1a90057b887d") : secret "maas-api-e2e-api-b-dad16f-serving-cert" not found
Jul 22 02:21:51.442508 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.442490    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d0eb9-4dbb-4166-838d-1a90057b887d-openshift-service-ca\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.462815 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.462793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn887\" (UniqueName: \"kubernetes.io/projected/8a8d0eb9-4dbb-4166-838d-1a90057b887d-kube-api-access-fn887\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.946595 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.946563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:51.948877 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:51.948855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls\") pod \"maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:52.120845 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:52.120815    2561 scope.go:117] "RemoveContainer" containerID="97f51f87d3838b096b5238238f43bc42bff8b92b140cb01682a94fbd937e6238"
Jul 22 02:21:52.196971 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:52.196915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:52.345560 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:52.345536    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"]
Jul 22 02:21:52.347620 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:21:52.347589    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8d0eb9_4dbb_4166_838d_1a90057b887d.slice/crio-fface55122e125b8ebdcfe9c84a8a76ee2e36a8fa67bfd8fac3b586fe0765621 WatchSource:0}: Error finding container fface55122e125b8ebdcfe9c84a8a76ee2e36a8fa67bfd8fac3b586fe0765621: Status 404 returned error can't find the container with id fface55122e125b8ebdcfe9c84a8a76ee2e36a8fa67bfd8fac3b586fe0765621
Jul 22 02:21:53.010812 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:53.010779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" event={"ID":"8a8d0eb9-4dbb-4166-838d-1a90057b887d","Type":"ContainerStarted","Data":"20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8"}
Jul 22 02:21:53.010996 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:53.010831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" event={"ID":"8a8d0eb9-4dbb-4166-838d-1a90057b887d","Type":"ContainerStarted","Data":"fface55122e125b8ebdcfe9c84a8a76ee2e36a8fa67bfd8fac3b586fe0765621"}
Jul 22 02:21:53.010996 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:53.010908    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:21:53.047654 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:53.047606    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" podStartSLOduration=1.6735113529999999 podStartE2EDuration="2.04759028s" totalImagesPullingTime="374.078927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:21:53.047287517 +0000 UTC m=+2224.550145745" watchObservedRunningTime="2026-07-22 02:21:53.04759028 +0000 UTC m=+2224.550448509"
Jul 22 02:21:59.021473 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:21:59.021442    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:22:03.809428 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:03.809388    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"]
Jul 22 02:22:03.812948 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:03.812931    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:03.815494 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:03.815474    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:22:03.821010 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:03.820986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"]
Jul 22 02:22:03.946741 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:03.946701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b096ab5e-93c8-4271-93aa-135e08414aa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-633172-rdx6d\" (UID: \"b096ab5e-93c8-4271-93aa-135e08414aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:04.048165 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:04.048128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b096ab5e-93c8-4271-93aa-135e08414aa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-633172-rdx6d\" (UID: \"b096ab5e-93c8-4271-93aa-135e08414aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:04.048770 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:04.048723    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b096ab5e-93c8-4271-93aa-135e08414aa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-633172-rdx6d\" (UID: \"b096ab5e-93c8-4271-93aa-135e08414aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:04.124498 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:04.124413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:04.261654 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:04.261586    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"]
Jul 22 02:22:04.263887 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:22:04.263860    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096ab5e_93c8_4271_93aa_135e08414aa3.slice/crio-96722b224463f918d96fe8dc63722ccdc6dab78f21c748d9328949a6a64908be WatchSource:0}: Error finding container 96722b224463f918d96fe8dc63722ccdc6dab78f21c748d9328949a6a64908be: Status 404 returned error can't find the container with id 96722b224463f918d96fe8dc63722ccdc6dab78f21c748d9328949a6a64908be
Jul 22 02:22:05.068888 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:05.068859    2561 generic.go:356] "Generic (PLEG): container finished" podID="b096ab5e-93c8-4271-93aa-135e08414aa3" containerID="f85ef4f7cb22f0e901650759a7c34b20c6283b3c0d55e6124af9df9f31d443ba" exitCode=0
Jul 22 02:22:05.069200 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:05.068942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d" event={"ID":"b096ab5e-93c8-4271-93aa-135e08414aa3","Type":"ContainerDied","Data":"f85ef4f7cb22f0e901650759a7c34b20c6283b3c0d55e6124af9df9f31d443ba"}
Jul 22 02:22:05.069200 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:05.068991    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d" event={"ID":"b096ab5e-93c8-4271-93aa-135e08414aa3","Type":"ContainerStarted","Data":"96722b224463f918d96fe8dc63722ccdc6dab78f21c748d9328949a6a64908be"}
Jul 22 02:22:06.214123 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:06.214091    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:06.365202 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:06.365120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b096ab5e-93c8-4271-93aa-135e08414aa3-maas-api-service-ca\") pod \"b096ab5e-93c8-4271-93aa-135e08414aa3\" (UID: \"b096ab5e-93c8-4271-93aa-135e08414aa3\") "
Jul 22 02:22:06.365442 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:06.365421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b096ab5e-93c8-4271-93aa-135e08414aa3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b096ab5e-93c8-4271-93aa-135e08414aa3" (UID: "b096ab5e-93c8-4271-93aa-135e08414aa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:22:06.466441 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:06.466403    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b096ab5e-93c8-4271-93aa-135e08414aa3-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:07.080597 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:07.080570    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"
Jul 22 02:22:07.080597 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:07.080592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d" event={"ID":"b096ab5e-93c8-4271-93aa-135e08414aa3","Type":"ContainerDied","Data":"96722b224463f918d96fe8dc63722ccdc6dab78f21c748d9328949a6a64908be"}
Jul 22 02:22:07.080835 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:07.080660    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96722b224463f918d96fe8dc63722ccdc6dab78f21c748d9328949a6a64908be"
Jul 22 02:22:18.996368 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:18.996277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"]
Jul 22 02:22:18.996947 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:18.996509    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b" podUID="e27ea217-ec2e-41f3-9062-632cc7e9a17e" containerName="maas-api" containerID="cri-o://f67e17a8325fc39a3cdf1d6920012427a2c3a05c35619309bf9e2ad888584554" gracePeriod=30
Jul 22 02:22:19.135281 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.135250    2561 generic.go:356] "Generic (PLEG): container finished" podID="e27ea217-ec2e-41f3-9062-632cc7e9a17e" containerID="f67e17a8325fc39a3cdf1d6920012427a2c3a05c35619309bf9e2ad888584554" exitCode=0
Jul 22 02:22:19.135425 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.135328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b" event={"ID":"e27ea217-ec2e-41f3-9062-632cc7e9a17e","Type":"ContainerDied","Data":"f67e17a8325fc39a3cdf1d6920012427a2c3a05c35619309bf9e2ad888584554"}
Jul 22 02:22:19.245165 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.245141    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:22:19.378332 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.378303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ea217-ec2e-41f3-9062-632cc7e9a17e-openshift-service-ca\") pod \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") "
Jul 22 02:22:19.378519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.378377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ea217-ec2e-41f3-9062-632cc7e9a17e-maas-api-tls\") pod \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") "
Jul 22 02:22:19.378519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.378417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn96j\" (UniqueName: \"kubernetes.io/projected/e27ea217-ec2e-41f3-9062-632cc7e9a17e-kube-api-access-fn96j\") pod \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\" (UID: \"e27ea217-ec2e-41f3-9062-632cc7e9a17e\") "
Jul 22 02:22:19.378747 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.378705    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e27ea217-ec2e-41f3-9062-632cc7e9a17e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e27ea217-ec2e-41f3-9062-632cc7e9a17e" (UID: "e27ea217-ec2e-41f3-9062-632cc7e9a17e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:22:19.380391 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.380363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e27ea217-ec2e-41f3-9062-632cc7e9a17e-kube-api-access-fn96j" (OuterVolumeSpecName: "kube-api-access-fn96j") pod "e27ea217-ec2e-41f3-9062-632cc7e9a17e" (UID: "e27ea217-ec2e-41f3-9062-632cc7e9a17e"). InnerVolumeSpecName "kube-api-access-fn96j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:22:19.380468 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.380454    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e27ea217-ec2e-41f3-9062-632cc7e9a17e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e27ea217-ec2e-41f3-9062-632cc7e9a17e" (UID: "e27ea217-ec2e-41f3-9062-632cc7e9a17e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:22:19.479004 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.478975    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn96j\" (UniqueName: \"kubernetes.io/projected/e27ea217-ec2e-41f3-9062-632cc7e9a17e-kube-api-access-fn96j\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:19.479004 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.478998    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e27ea217-ec2e-41f3-9062-632cc7e9a17e-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:19.479158 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:19.479008    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e27ea217-ec2e-41f3-9062-632cc7e9a17e-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:20.142241 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:20.142208    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"
Jul 22 02:22:20.142623 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:20.142260    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b" event={"ID":"e27ea217-ec2e-41f3-9062-632cc7e9a17e","Type":"ContainerDied","Data":"8b4039263dca23b4272b054adee18cb37fb0c67808061f62d19e1ed3d0adfc4b"}
Jul 22 02:22:20.142623 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:20.142329    2561 scope.go:117] "RemoveContainer" containerID="f67e17a8325fc39a3cdf1d6920012427a2c3a05c35619309bf9e2ad888584554"
Jul 22 02:22:20.169203 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:20.169167    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"]
Jul 22 02:22:20.173095 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:20.173072    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-633172-7fc4c9cc6d-l455b"]
Jul 22 02:22:21.101859 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:21.101825    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e27ea217-ec2e-41f3-9062-632cc7e9a17e" path="/var/lib/kubelet/pods/e27ea217-ec2e-41f3-9062-632cc7e9a17e/volumes"
Jul 22 02:22:23.912408 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:23.912374    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"]
Jul 22 02:22:23.915390 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:23.915357    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-633172-rdx6d"]
Jul 22 02:22:25.102053 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:25.102023    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b096ab5e-93c8-4271-93aa-135e08414aa3" path="/var/lib/kubelet/pods/b096ab5e-93c8-4271-93aa-135e08414aa3/volumes"
Jul 22 02:22:31.171919 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.171888    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"]
Jul 22 02:22:31.172305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.172210    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b096ab5e-93c8-4271-93aa-135e08414aa3" containerName="revoke-keys"
Jul 22 02:22:31.172305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.172220    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b096ab5e-93c8-4271-93aa-135e08414aa3" containerName="revoke-keys"
Jul 22 02:22:31.172305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.172232    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e27ea217-ec2e-41f3-9062-632cc7e9a17e" containerName="maas-api"
Jul 22 02:22:31.172305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.172238    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e27ea217-ec2e-41f3-9062-632cc7e9a17e" containerName="maas-api"
Jul 22 02:22:31.172305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.172291    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e27ea217-ec2e-41f3-9062-632cc7e9a17e" containerName="maas-api"
Jul 22 02:22:31.172305 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.172299    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b096ab5e-93c8-4271-93aa-135e08414aa3" containerName="revoke-keys"
Jul 22 02:22:31.175223 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.175206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:31.178383 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.178363    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:22:31.186030 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.186005    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"]
Jul 22 02:22:31.275804 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.275775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a70407-6535-4baa-a7bd-519d057abb75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd\" (UID: \"13a70407-6535-4baa-a7bd-519d057abb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:31.376233 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.376201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a70407-6535-4baa-a7bd-519d057abb75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd\" (UID: \"13a70407-6535-4baa-a7bd-519d057abb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:31.376835 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.376816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a70407-6535-4baa-a7bd-519d057abb75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd\" (UID: \"13a70407-6535-4baa-a7bd-519d057abb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:31.485709 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.485605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:31.614813 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:31.614788    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"]
Jul 22 02:22:31.616712 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:22:31.616689    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a70407_6535_4baa_a7bd_519d057abb75.slice/crio-0ca3d1be1320736f905b7d14d69e8528d12d14804cb332b161e0502978f263be WatchSource:0}: Error finding container 0ca3d1be1320736f905b7d14d69e8528d12d14804cb332b161e0502978f263be: Status 404 returned error can't find the container with id 0ca3d1be1320736f905b7d14d69e8528d12d14804cb332b161e0502978f263be
Jul 22 02:22:32.199412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:32.199325    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd" event={"ID":"13a70407-6535-4baa-a7bd-519d057abb75","Type":"ContainerStarted","Data":"061e52b000eee3d1ba977ed29c806bead51274a456e6f0eab769d9696deff01d"}
Jul 22 02:22:32.199412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:32.199382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd" event={"ID":"13a70407-6535-4baa-a7bd-519d057abb75","Type":"ContainerStarted","Data":"0ca3d1be1320736f905b7d14d69e8528d12d14804cb332b161e0502978f263be"}
Jul 22 02:22:32.215238 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:32.215191    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd" podStartSLOduration=1.2151765509999999 podStartE2EDuration="1.215176551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:22:32.214877442 +0000 UTC m=+2263.717735671" watchObservedRunningTime="2026-07-22 02:22:32.215176551 +0000 UTC m=+2263.718034780"
Jul 22 02:22:33.204718 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:33.204681    2561 generic.go:356] "Generic (PLEG): container finished" podID="13a70407-6535-4baa-a7bd-519d057abb75" containerID="061e52b000eee3d1ba977ed29c806bead51274a456e6f0eab769d9696deff01d" exitCode=0
Jul 22 02:22:33.205115 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:33.204760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd" event={"ID":"13a70407-6535-4baa-a7bd-519d057abb75","Type":"ContainerDied","Data":"061e52b000eee3d1ba977ed29c806bead51274a456e6f0eab769d9696deff01d"}
Jul 22 02:22:34.339168 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:34.339139    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:34.398818 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:34.398791    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a70407-6535-4baa-a7bd-519d057abb75-maas-api-service-ca\") pod \"13a70407-6535-4baa-a7bd-519d057abb75\" (UID: \"13a70407-6535-4baa-a7bd-519d057abb75\") "
Jul 22 02:22:34.399104 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:34.399081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a70407-6535-4baa-a7bd-519d057abb75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13a70407-6535-4baa-a7bd-519d057abb75" (UID: "13a70407-6535-4baa-a7bd-519d057abb75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:22:34.500346 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:34.500273    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a70407-6535-4baa-a7bd-519d057abb75-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:35.215948 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:35.215916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"
Jul 22 02:22:35.215948 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:35.215928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd" event={"ID":"13a70407-6535-4baa-a7bd-519d057abb75","Type":"ContainerDied","Data":"0ca3d1be1320736f905b7d14d69e8528d12d14804cb332b161e0502978f263be"}
Jul 22 02:22:35.216134 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:35.215974    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca3d1be1320736f905b7d14d69e8528d12d14804cb332b161e0502978f263be"
Jul 22 02:22:46.763710 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:46.763673    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"]
Jul 22 02:22:46.764185 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:46.763930    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" podUID="8a8d0eb9-4dbb-4166-838d-1a90057b887d" containerName="maas-api" containerID="cri-o://20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8" gracePeriod=30
Jul 22 02:22:47.009457 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.009436    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:22:47.100546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.100460    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls\") pod \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") "
Jul 22 02:22:47.100546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.100503    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d0eb9-4dbb-4166-838d-1a90057b887d-openshift-service-ca\") pod \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") "
Jul 22 02:22:47.100546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.100528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn887\" (UniqueName: \"kubernetes.io/projected/8a8d0eb9-4dbb-4166-838d-1a90057b887d-kube-api-access-fn887\") pod \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\" (UID: \"8a8d0eb9-4dbb-4166-838d-1a90057b887d\") "
Jul 22 02:22:47.100909 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.100883    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a8d0eb9-4dbb-4166-838d-1a90057b887d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8a8d0eb9-4dbb-4166-838d-1a90057b887d" (UID: "8a8d0eb9-4dbb-4166-838d-1a90057b887d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:22:47.102873 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.102847    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8d0eb9-4dbb-4166-838d-1a90057b887d-kube-api-access-fn887" (OuterVolumeSpecName: "kube-api-access-fn887") pod "8a8d0eb9-4dbb-4166-838d-1a90057b887d" (UID: "8a8d0eb9-4dbb-4166-838d-1a90057b887d"). InnerVolumeSpecName "kube-api-access-fn887". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:22:47.103150 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.103128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8a8d0eb9-4dbb-4166-838d-1a90057b887d" (UID: "8a8d0eb9-4dbb-4166-838d-1a90057b887d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:22:47.201554 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.201515    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a8d0eb9-4dbb-4166-838d-1a90057b887d-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:47.201767 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.201571    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8d0eb9-4dbb-4166-838d-1a90057b887d-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:47.201767 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.201583    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn887\" (UniqueName: \"kubernetes.io/projected/8a8d0eb9-4dbb-4166-838d-1a90057b887d-kube-api-access-fn887\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:22:47.272378 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.272345    2561 generic.go:356] "Generic (PLEG): container finished" podID="8a8d0eb9-4dbb-4166-838d-1a90057b887d" containerID="20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8" exitCode=0
Jul 22 02:22:47.272558 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.272396    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"
Jul 22 02:22:47.272558 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.272424    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" event={"ID":"8a8d0eb9-4dbb-4166-838d-1a90057b887d","Type":"ContainerDied","Data":"20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8"}
Jul 22 02:22:47.272558 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.272476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf" event={"ID":"8a8d0eb9-4dbb-4166-838d-1a90057b887d","Type":"ContainerDied","Data":"fface55122e125b8ebdcfe9c84a8a76ee2e36a8fa67bfd8fac3b586fe0765621"}
Jul 22 02:22:47.272558 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.272499    2561 scope.go:117] "RemoveContainer" containerID="20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8"
Jul 22 02:22:47.281115 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.281097    2561 scope.go:117] "RemoveContainer" containerID="20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8"
Jul 22 02:22:47.281379 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:22:47.281359    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8\": container with ID starting with 20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8 not found: ID does not exist" containerID="20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8"
Jul 22 02:22:47.281418 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.281388    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8"} err="failed to get container status \"20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8\": rpc error: code = NotFound desc = could not find container \"20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8\": container with ID starting with 20ec7e5083d86ef60d15d931a8dcf8470657587d94d229ba46178b799e6ddfc8 not found: ID does not exist"
Jul 22 02:22:47.297996 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.297967    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"]
Jul 22 02:22:47.300268 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:47.300247    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dad16f-dddf5cb49-vgcgf"]
Jul 22 02:22:49.101843 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:49.101811    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a8d0eb9-4dbb-4166-838d-1a90057b887d" path="/var/lib/kubelet/pods/8a8d0eb9-4dbb-4166-838d-1a90057b887d/volumes"
Jul 22 02:22:51.243015 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:51.242972    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"]
Jul 22 02:22:51.245121 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:51.245095    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dad16f-vq7pd"]
Jul 22 02:22:53.101427 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:22:53.101397    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a70407-6535-4baa-a7bd-519d057abb75" path="/var/lib/kubelet/pods/13a70407-6535-4baa-a7bd-519d057abb75/volumes"
Jul 22 02:23:02.595171 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595137    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"]
Jul 22 02:23:02.595559 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a8d0eb9-4dbb-4166-838d-1a90057b887d" containerName="maas-api"
Jul 22 02:23:02.595559 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595483    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a8d0eb9-4dbb-4166-838d-1a90057b887d" containerName="maas-api"
Jul 22 02:23:02.595559 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595500    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a70407-6535-4baa-a7bd-519d057abb75" containerName="revoke-keys"
Jul 22 02:23:02.595559 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595505    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a70407-6535-4baa-a7bd-519d057abb75" containerName="revoke-keys"
Jul 22 02:23:02.595690 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595561    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a70407-6535-4baa-a7bd-519d057abb75" containerName="revoke-keys"
Jul 22 02:23:02.595690 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.595570    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a8d0eb9-4dbb-4166-838d-1a90057b887d" containerName="maas-api"
Jul 22 02:23:02.598388 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.598373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.601121 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.601095    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-000830-serving-cert\""
Jul 22 02:23:02.601271 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.601178    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zzw76\""
Jul 22 02:23:02.612966 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.612936    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"]
Jul 22 02:23:02.732719 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.732680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.732719 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.732717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crnxr\" (UniqueName: \"kubernetes.io/projected/869caafd-43de-44b2-acd1-285902050748-kube-api-access-crnxr\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.732963 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.732812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/869caafd-43de-44b2-acd1-285902050748-openshift-service-ca\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.833694 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.833655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.833694 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.833692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crnxr\" (UniqueName: \"kubernetes.io/projected/869caafd-43de-44b2-acd1-285902050748-kube-api-access-crnxr\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.833926 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.833747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/869caafd-43de-44b2-acd1-285902050748-openshift-service-ca\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.833926 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:23:02.833839    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-serving-cert: secret "maas-api-e2e-shared-a-000830-serving-cert" not found
Jul 22 02:23:02.833993 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:23:02.833926    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls podName:869caafd-43de-44b2-acd1-285902050748 nodeName:}" failed. No retries permitted until 2026-07-22 02:23:03.333904828 +0000 UTC m=+2294.836763041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls") pod "maas-api-e2e-shared-a-000830-8454c77556-cm9f5" (UID: "869caafd-43de-44b2-acd1-285902050748") : secret "maas-api-e2e-shared-a-000830-serving-cert" not found
Jul 22 02:23:02.834412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.834396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/869caafd-43de-44b2-acd1-285902050748-openshift-service-ca\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:02.846270 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:02.846202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crnxr\" (UniqueName: \"kubernetes.io/projected/869caafd-43de-44b2-acd1-285902050748-kube-api-access-crnxr\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:03.339117 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:03.339078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:03.341927 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:03.341897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls\") pod \"maas-api-e2e-shared-a-000830-8454c77556-cm9f5\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:03.509134 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:03.509098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:03.647975 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:03.647951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"]
Jul 22 02:23:03.650276 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:23:03.650247    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869caafd_43de_44b2_acd1_285902050748.slice/crio-90cd8a2d5b5b158908ec35b8a0f889e97707d7fe887020a13c503446cac028c5 WatchSource:0}: Error finding container 90cd8a2d5b5b158908ec35b8a0f889e97707d7fe887020a13c503446cac028c5: Status 404 returned error can't find the container with id 90cd8a2d5b5b158908ec35b8a0f889e97707d7fe887020a13c503446cac028c5
Jul 22 02:23:04.353536 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:04.353446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5" event={"ID":"869caafd-43de-44b2-acd1-285902050748","Type":"ContainerStarted","Data":"88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82"}
Jul 22 02:23:04.353536 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:04.353501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5" event={"ID":"869caafd-43de-44b2-acd1-285902050748","Type":"ContainerStarted","Data":"90cd8a2d5b5b158908ec35b8a0f889e97707d7fe887020a13c503446cac028c5"}
Jul 22 02:23:04.353757 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:04.353562    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:04.373100 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:04.373041    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5" podStartSLOduration=1.991869729 podStartE2EDuration="2.373024985s" totalImagesPullingTime="381.155256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:23:04.371433669 +0000 UTC m=+2295.874291898" watchObservedRunningTime="2026-07-22 02:23:04.373024985 +0000 UTC m=+2295.875883213"
Jul 22 02:23:10.362570 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.362542    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:23:10.745203 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.745167    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"]
Jul 22 02:23:10.749206 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.749185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:10.751721 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.751698    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-7eead4-serving-cert\""
Jul 22 02:23:10.767614 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.767584    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"]
Jul 22 02:23:10.905778 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.905723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810c3de-71b4-4783-adc6-7cc13f273b4c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:10.905960 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.905820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pkm\" (UniqueName: \"kubernetes.io/projected/c810c3de-71b4-4783-adc6-7cc13f273b4c-kube-api-access-v2pkm\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:10.905960 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:10.905839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.006637 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.006609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810c3de-71b4-4783-adc6-7cc13f273b4c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.006842 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.006676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pkm\" (UniqueName: \"kubernetes.io/projected/c810c3de-71b4-4783-adc6-7cc13f273b4c-kube-api-access-v2pkm\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.006842 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.006696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.006842 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:23:11.006808    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-serving-cert: secret "maas-api-e2e-shared-b-7eead4-serving-cert" not found
Jul 22 02:23:11.007015 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:23:11.006864    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls podName:c810c3de-71b4-4783-adc6-7cc13f273b4c nodeName:}" failed. No retries permitted until 2026-07-22 02:23:11.506849258 +0000 UTC m=+2303.009707465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls") pod "maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" (UID: "c810c3de-71b4-4783-adc6-7cc13f273b4c") : secret "maas-api-e2e-shared-b-7eead4-serving-cert" not found
Jul 22 02:23:11.007266 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.007248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810c3de-71b4-4783-adc6-7cc13f273b4c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.016142 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.016112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pkm\" (UniqueName: \"kubernetes.io/projected/c810c3de-71b4-4783-adc6-7cc13f273b4c-kube-api-access-v2pkm\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.512550 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.512517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.514845 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.514823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.660132 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.660076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:11.795368 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:11.795343    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"]
Jul 22 02:23:11.797795 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:23:11.797770    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc810c3de_71b4_4783_adc6_7cc13f273b4c.slice/crio-009ce7c685edf762f2fc3c579147f316d31ccfb6a7e4d441eaad975a54987b05 WatchSource:0}: Error finding container 009ce7c685edf762f2fc3c579147f316d31ccfb6a7e4d441eaad975a54987b05: Status 404 returned error can't find the container with id 009ce7c685edf762f2fc3c579147f316d31ccfb6a7e4d441eaad975a54987b05
Jul 22 02:23:12.391972 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:12.391934    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" event={"ID":"c810c3de-71b4-4783-adc6-7cc13f273b4c","Type":"ContainerStarted","Data":"fe81b7eccfb48f5c093c4567a982a5d93fa287c404ac304aed60d45bd58a01f6"}
Jul 22 02:23:12.392148 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:12.392014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" event={"ID":"c810c3de-71b4-4783-adc6-7cc13f273b4c","Type":"ContainerStarted","Data":"009ce7c685edf762f2fc3c579147f316d31ccfb6a7e4d441eaad975a54987b05"}
Jul 22 02:23:12.392148 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:12.392067    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:12.411132 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:12.411021    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" podStartSLOduration=2.065137807 podStartE2EDuration="2.411004868s" totalImagesPullingTime="345.867061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:23:12.409637879 +0000 UTC m=+2303.912496109" watchObservedRunningTime="2026-07-22 02:23:12.411004868 +0000 UTC m=+2303.913863100"
Jul 22 02:23:18.401540 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:18.401507    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:23:26.698942 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.698911    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"]
Jul 22 02:23:26.702327 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.702306    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.705067 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.705037    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"openshift-service-ca.crt\""
Jul 22 02:23:26.706170 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.706148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"auth-test-model-000830-kserve-self-signed-certs\""
Jul 22 02:23:26.706273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.706252    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"default-dockercfg-8jfbh\""
Jul 22 02:23:26.706357 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.706257    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"kube-root-ca.crt\""
Jul 22 02:23:26.713769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.713708    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"]
Jul 22 02:23:26.740222 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.740184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-home\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.740389 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.740233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-model-cache\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.740389 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.740283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764mt\" (UniqueName: \"kubernetes.io/projected/03188910-d4ea-4fd2-acc0-7c0452368672-kube-api-access-764mt\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.740389 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.740371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-dshm\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.740539 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.740398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-tmp-dir\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.740539 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.740421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188910-d4ea-4fd2-acc0-7c0452368672-tls-certs\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.840920 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.840877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-dshm\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.840920 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.840925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-tmp-dir\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841176 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.840993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188910-d4ea-4fd2-acc0-7c0452368672-tls-certs\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841176 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.841111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-home\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841176 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.841142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-model-cache\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841405 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.841180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764mt\" (UniqueName: \"kubernetes.io/projected/03188910-d4ea-4fd2-acc0-7c0452368672-kube-api-access-764mt\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841405 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.841376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-tmp-dir\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841610 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.841559    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-home\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.841772 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.841639    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-model-cache\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.843218 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.843197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-dshm\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.843565 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.843534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188910-d4ea-4fd2-acc0-7c0452368672-tls-certs\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:26.852051 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:26.852028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764mt\" (UniqueName: \"kubernetes.io/projected/03188910-d4ea-4fd2-acc0-7c0452368672-kube-api-access-764mt\") pod \"auth-test-model-000830-kserve-8bbbd6fb7-ksmxc\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") " pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:27.012827 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:27.012795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:27.145466 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:27.145435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"]
Jul 22 02:23:27.148395 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:23:27.148371    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03188910_d4ea_4fd2_acc0_7c0452368672.slice/crio-4654489f3f9387d574a8831c80921499f377c4ee8c1c5187e2eedb5df1bf946e WatchSource:0}: Error finding container 4654489f3f9387d574a8831c80921499f377c4ee8c1c5187e2eedb5df1bf946e: Status 404 returned error can't find the container with id 4654489f3f9387d574a8831c80921499f377c4ee8c1c5187e2eedb5df1bf946e
Jul 22 02:23:27.465543 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:27.465457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc" event={"ID":"03188910-d4ea-4fd2-acc0-7c0452368672","Type":"ContainerStarted","Data":"20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b"}
Jul 22 02:23:27.465543 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:27.465501    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc" event={"ID":"03188910-d4ea-4fd2-acc0-7c0452368672","Type":"ContainerStarted","Data":"4654489f3f9387d574a8831c80921499f377c4ee8c1c5187e2eedb5df1bf946e"}
Jul 22 02:23:27.483111 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:27.483064    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc" podStartSLOduration=1.483051622 podStartE2EDuration="1.483051622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:23:27.482020859 +0000 UTC m=+2318.984879091" watchObservedRunningTime="2026-07-22 02:23:27.483051622 +0000 UTC m=+2318.985909901"
Jul 22 02:23:37.013187 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:37.013149    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:37.013618 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:37.013205    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:37.025929 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:37.025901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:23:37.525172 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:23:37.525145    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:24:01.111817 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.111712    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"]
Jul 22 02:24:01.115017 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.115000    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.118837 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.118807    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"default-dockercfg-h2zwx\""
Jul 22 02:24:01.118981 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.118811    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"openshift-service-ca.crt\""
Jul 22 02:24:01.118981 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.118939    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"kube-root-ca.crt\""
Jul 22 02:24:01.118981 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.118961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"auth-test-model-7eead4-kserve-self-signed-certs\""
Jul 22 02:24:01.125742 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.125703    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"]
Jul 22 02:24:01.226751 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.226700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-dshm\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.226940 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.226778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-tmp-dir\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.226940 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.226817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-home\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.226940 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.226836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-model-cache\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.226940 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.226890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b22f4b-1f7a-4a22-b633-53d94f304599-tls-certs\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.227090 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.226939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p74v\" (UniqueName: \"kubernetes.io/projected/16b22f4b-1f7a-4a22-b633-53d94f304599-kube-api-access-2p74v\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.327555 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-tmp-dir\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.327769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327567    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-home\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.327769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-model-cache\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.327769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b22f4b-1f7a-4a22-b633-53d94f304599-tls-certs\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.327769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p74v\" (UniqueName: \"kubernetes.io/projected/16b22f4b-1f7a-4a22-b633-53d94f304599-kube-api-access-2p74v\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.327769 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-dshm\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.328063 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.327941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-tmp-dir\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.328063 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.328002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-home\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.328063 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.328025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-model-cache\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.329869 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.329849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-dshm\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.330229 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.330209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b22f4b-1f7a-4a22-b633-53d94f304599-tls-certs\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.335878 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.335854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p74v\" (UniqueName: \"kubernetes.io/projected/16b22f4b-1f7a-4a22-b633-53d94f304599-kube-api-access-2p74v\") pod \"auth-test-model-7eead4-kserve-67967969f5-56zhp\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") " pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.424694 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.424609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:01.581390 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.581363    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"]
Jul 22 02:24:01.583575 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:24:01.583552    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16b22f4b_1f7a_4a22_b633_53d94f304599.slice/crio-ac64dd5fc6c1f3252a0bf528c4908a2eef8972f8ee1d2832a671143db5e2be4e WatchSource:0}: Error finding container ac64dd5fc6c1f3252a0bf528c4908a2eef8972f8ee1d2832a671143db5e2be4e: Status 404 returned error can't find the container with id ac64dd5fc6c1f3252a0bf528c4908a2eef8972f8ee1d2832a671143db5e2be4e
Jul 22 02:24:01.627853 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:01.627826    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp" event={"ID":"16b22f4b-1f7a-4a22-b633-53d94f304599","Type":"ContainerStarted","Data":"ac64dd5fc6c1f3252a0bf528c4908a2eef8972f8ee1d2832a671143db5e2be4e"}
Jul 22 02:24:02.636664 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:02.636625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp" event={"ID":"16b22f4b-1f7a-4a22-b633-53d94f304599","Type":"ContainerStarted","Data":"ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9"}
Jul 22 02:24:02.654546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:02.654493    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp" podStartSLOduration=1.654476974 podStartE2EDuration="1.654476974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:24:02.652037465 +0000 UTC m=+2354.154895695" watchObservedRunningTime="2026-07-22 02:24:02.654476974 +0000 UTC m=+2354.157335203"
Jul 22 02:24:11.425317 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:11.425273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:11.425821 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:11.425336    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:11.438333 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:11.438306    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:11.689377 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:11.689304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:24:49.156316 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:49.156289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:24:49.157478 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:24:49.157459    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:26:04.738260 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:04.738226    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"]
Jul 22 02:26:05.641503 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:05.641463    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"]
Jul 22 02:26:19.758429 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:19.758383    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc" podUID="03188910-d4ea-4fd2-acc0-7c0452368672" containerName="main" containerID="cri-o://20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b" gracePeriod=45
Jul 22 02:26:19.998702 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:19.998676    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:26:20.034635 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034557    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188910-d4ea-4fd2-acc0-7c0452368672-tls-certs\") pod \"03188910-d4ea-4fd2-acc0-7c0452368672\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") "
Jul 22 02:26:20.034635 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-tmp-dir\") pod \"03188910-d4ea-4fd2-acc0-7c0452368672\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") "
Jul 22 02:26:20.034886 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-home\") pod \"03188910-d4ea-4fd2-acc0-7c0452368672\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") "
Jul 22 02:26:20.034886 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034687    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-model-cache\") pod \"03188910-d4ea-4fd2-acc0-7c0452368672\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") "
Jul 22 02:26:20.034886 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034713    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-dshm\") pod \"03188910-d4ea-4fd2-acc0-7c0452368672\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") "
Jul 22 02:26:20.034886 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-764mt\" (UniqueName: \"kubernetes.io/projected/03188910-d4ea-4fd2-acc0-7c0452368672-kube-api-access-764mt\") pod \"03188910-d4ea-4fd2-acc0-7c0452368672\" (UID: \"03188910-d4ea-4fd2-acc0-7c0452368672\") "
Jul 22 02:26:20.035097 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-home" (OuterVolumeSpecName: "home") pod "03188910-d4ea-4fd2-acc0-7c0452368672" (UID: "03188910-d4ea-4fd2-acc0-7c0452368672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.035097 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03188910-d4ea-4fd2-acc0-7c0452368672" (UID: "03188910-d4ea-4fd2-acc0-7c0452368672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.035097 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.034999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-model-cache" (OuterVolumeSpecName: "model-cache") pod "03188910-d4ea-4fd2-acc0-7c0452368672" (UID: "03188910-d4ea-4fd2-acc0-7c0452368672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.035097 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.035036    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:20.035097 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.035053    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:20.037404 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.037365    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03188910-d4ea-4fd2-acc0-7c0452368672-kube-api-access-764mt" (OuterVolumeSpecName: "kube-api-access-764mt") pod "03188910-d4ea-4fd2-acc0-7c0452368672" (UID: "03188910-d4ea-4fd2-acc0-7c0452368672"). InnerVolumeSpecName "kube-api-access-764mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:26:20.037513 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.037477    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03188910-d4ea-4fd2-acc0-7c0452368672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03188910-d4ea-4fd2-acc0-7c0452368672" (UID: "03188910-d4ea-4fd2-acc0-7c0452368672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:26:20.037650 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.037631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-dshm" (OuterVolumeSpecName: "dshm") pod "03188910-d4ea-4fd2-acc0-7c0452368672" (UID: "03188910-d4ea-4fd2-acc0-7c0452368672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.136534 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.136500    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:20.136534 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.136540    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03188910-d4ea-4fd2-acc0-7c0452368672-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:20.136766 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.136553    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-764mt\" (UniqueName: \"kubernetes.io/projected/03188910-d4ea-4fd2-acc0-7c0452368672-kube-api-access-764mt\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:20.136766 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.136562    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03188910-d4ea-4fd2-acc0-7c0452368672-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:20.295345 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.295258    2561 generic.go:356] "Generic (PLEG): container finished" podID="03188910-d4ea-4fd2-acc0-7c0452368672" containerID="20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b" exitCode=0
Jul 22 02:26:20.295345 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.295300    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"
Jul 22 02:26:20.295530 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.295344    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc" event={"ID":"03188910-d4ea-4fd2-acc0-7c0452368672","Type":"ContainerDied","Data":"20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b"}
Jul 22 02:26:20.295530 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.295390    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc" event={"ID":"03188910-d4ea-4fd2-acc0-7c0452368672","Type":"ContainerDied","Data":"4654489f3f9387d574a8831c80921499f377c4ee8c1c5187e2eedb5df1bf946e"}
Jul 22 02:26:20.295530 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.295410    2561 scope.go:117] "RemoveContainer" containerID="20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b"
Jul 22 02:26:20.304367 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.304348    2561 scope.go:117] "RemoveContainer" containerID="20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b"
Jul 22 02:26:20.304643 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:26:20.304624    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b\": container with ID starting with 20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b not found: ID does not exist" containerID="20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b"
Jul 22 02:26:20.304678 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.304652    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b"} err="failed to get container status \"20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b\": rpc error: code = NotFound desc = could not find container \"20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b\": container with ID starting with 20e66cff6e4f54825e7cd7e51f3d64bc0ef66fc2a1c58e7b8aeafac003824c8b not found: ID does not exist"
Jul 22 02:26:20.316453 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.316430    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"]
Jul 22 02:26:20.321170 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.321149    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-000830/auth-test-model-000830-kserve-8bbbd6fb7-ksmxc"]
Jul 22 02:26:20.661560 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.661474    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp" podUID="16b22f4b-1f7a-4a22-b633-53d94f304599" containerName="main" containerID="cri-o://ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9" gracePeriod=45
Jul 22 02:26:20.905068 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.905047    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:26:20.944639 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.944562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p74v\" (UniqueName: \"kubernetes.io/projected/16b22f4b-1f7a-4a22-b633-53d94f304599-kube-api-access-2p74v\") pod \"16b22f4b-1f7a-4a22-b633-53d94f304599\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") "
Jul 22 02:26:20.944639 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.944598    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-model-cache\") pod \"16b22f4b-1f7a-4a22-b633-53d94f304599\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") "
Jul 22 02:26:20.944639 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.944617    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-home\") pod \"16b22f4b-1f7a-4a22-b633-53d94f304599\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") "
Jul 22 02:26:20.944924 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.944650    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b22f4b-1f7a-4a22-b633-53d94f304599-tls-certs\") pod \"16b22f4b-1f7a-4a22-b633-53d94f304599\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") "
Jul 22 02:26:20.944924 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.944771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-dshm\") pod \"16b22f4b-1f7a-4a22-b633-53d94f304599\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") "
Jul 22 02:26:20.944924 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.944802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-tmp-dir\") pod \"16b22f4b-1f7a-4a22-b633-53d94f304599\" (UID: \"16b22f4b-1f7a-4a22-b633-53d94f304599\") "
Jul 22 02:26:20.945185 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.945139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-model-cache" (OuterVolumeSpecName: "model-cache") pod "16b22f4b-1f7a-4a22-b633-53d94f304599" (UID: "16b22f4b-1f7a-4a22-b633-53d94f304599"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.945185 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.945172    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-home" (OuterVolumeSpecName: "home") pod "16b22f4b-1f7a-4a22-b633-53d94f304599" (UID: "16b22f4b-1f7a-4a22-b633-53d94f304599"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.945349 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.945245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16b22f4b-1f7a-4a22-b633-53d94f304599" (UID: "16b22f4b-1f7a-4a22-b633-53d94f304599"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.947087 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.947054    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-dshm" (OuterVolumeSpecName: "dshm") pod "16b22f4b-1f7a-4a22-b633-53d94f304599" (UID: "16b22f4b-1f7a-4a22-b633-53d94f304599"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:26:20.947226 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.947209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b22f4b-1f7a-4a22-b633-53d94f304599-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16b22f4b-1f7a-4a22-b633-53d94f304599" (UID: "16b22f4b-1f7a-4a22-b633-53d94f304599"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:26:20.947358 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:20.947339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b22f4b-1f7a-4a22-b633-53d94f304599-kube-api-access-2p74v" (OuterVolumeSpecName: "kube-api-access-2p74v") pod "16b22f4b-1f7a-4a22-b633-53d94f304599" (UID: "16b22f4b-1f7a-4a22-b633-53d94f304599"). InnerVolumeSpecName "kube-api-access-2p74v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:26:21.045888 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.045849    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p74v\" (UniqueName: \"kubernetes.io/projected/16b22f4b-1f7a-4a22-b633-53d94f304599-kube-api-access-2p74v\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:21.045888 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.045884    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:21.046082 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.045898    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:21.046082 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.045909    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16b22f4b-1f7a-4a22-b633-53d94f304599-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:21.046082 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.045921    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:21.046082 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.045933    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b22f4b-1f7a-4a22-b633-53d94f304599-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:26:21.101963 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.101931    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03188910-d4ea-4fd2-acc0-7c0452368672" path="/var/lib/kubelet/pods/03188910-d4ea-4fd2-acc0-7c0452368672/volumes"
Jul 22 02:26:21.302129 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.302096    2561 generic.go:356] "Generic (PLEG): container finished" podID="16b22f4b-1f7a-4a22-b633-53d94f304599" containerID="ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9" exitCode=0
Jul 22 02:26:21.302298 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.302150    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"
Jul 22 02:26:21.302298 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.302177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp" event={"ID":"16b22f4b-1f7a-4a22-b633-53d94f304599","Type":"ContainerDied","Data":"ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9"}
Jul 22 02:26:21.302298 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.302226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp" event={"ID":"16b22f4b-1f7a-4a22-b633-53d94f304599","Type":"ContainerDied","Data":"ac64dd5fc6c1f3252a0bf528c4908a2eef8972f8ee1d2832a671143db5e2be4e"}
Jul 22 02:26:21.302298 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.302243    2561 scope.go:117] "RemoveContainer" containerID="ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9"
Jul 22 02:26:21.310673 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.310654    2561 scope.go:117] "RemoveContainer" containerID="ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9"
Jul 22 02:26:21.311012 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:26:21.310991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9\": container with ID starting with ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9 not found: ID does not exist" containerID="ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9"
Jul 22 02:26:21.311109 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.311018    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9"} err="failed to get container status \"ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9\": rpc error: code = NotFound desc = could not find container \"ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9\": container with ID starting with ea1d9850e0a68c4dfffd5279d7e99233194f13706accc1d5eae463ce8b2488a9 not found: ID does not exist"
Jul 22 02:26:21.336041 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.336019    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"]
Jul 22 02:26:21.342394 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:21.342370    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/auth-test-model-7eead4-kserve-67967969f5-56zhp"]
Jul 22 02:26:23.102865 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:23.102822    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16b22f4b-1f7a-4a22-b633-53d94f304599" path="/var/lib/kubelet/pods/16b22f4b-1f7a-4a22-b633-53d94f304599/volumes"
Jul 22 02:26:40.394319 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394246    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"]
Jul 22 02:26:40.396711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394822    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16b22f4b-1f7a-4a22-b633-53d94f304599" containerName="main"
Jul 22 02:26:40.396711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="16b22f4b-1f7a-4a22-b633-53d94f304599" containerName="main"
Jul 22 02:26:40.396711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03188910-d4ea-4fd2-acc0-7c0452368672" containerName="main"
Jul 22 02:26:40.396711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394864    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="03188910-d4ea-4fd2-acc0-7c0452368672" containerName="main"
Jul 22 02:26:40.396711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394944    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="03188910-d4ea-4fd2-acc0-7c0452368672" containerName="main"
Jul 22 02:26:40.396711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.394955    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="16b22f4b-1f7a-4a22-b633-53d94f304599" containerName="main"
Jul 22 02:26:40.398072 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.398051    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.401710 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.401678    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"default-dockercfg-h2zwx\""
Jul 22 02:26:40.401857 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.401715    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"openshift-service-ca.crt\""
Jul 22 02:26:40.401857 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.401686    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"kube-root-ca.crt\""
Jul 22 02:26:40.401857 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.401677    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"sub-test-model-7eead4-kserve-self-signed-certs\""
Jul 22 02:26:40.406995 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.406973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"]
Jul 22 02:26:40.513081 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.513050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57kr\" (UniqueName: \"kubernetes.io/projected/8fea78ed-30b1-4b6a-a510-0924a507b5f9-kube-api-access-t57kr\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.513259 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.513093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tls-certs\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.513259 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.513117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tmp-dir\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.513259 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.513185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-dshm\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.513259 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.513212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-model-cache\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.513259 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.513241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-home\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614404 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-home\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614568 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57kr\" (UniqueName: \"kubernetes.io/projected/8fea78ed-30b1-4b6a-a510-0924a507b5f9-kube-api-access-t57kr\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614568 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tls-certs\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614568 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tmp-dir\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614568 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-dshm\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614568 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-model-cache\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614872 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-home\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.614953 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tmp-dir\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.615001 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.614970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-model-cache\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.616780 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.616763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-dshm\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.616847 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.616838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tls-certs\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.622237 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.622218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57kr\" (UniqueName: \"kubernetes.io/projected/8fea78ed-30b1-4b6a-a510-0924a507b5f9-kube-api-access-t57kr\") pod \"sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") " pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.710230 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.710133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:40.847970 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:40.847941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"]
Jul 22 02:26:40.850165 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:26:40.850127    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fea78ed_30b1_4b6a_a510_0924a507b5f9.slice/crio-cdebad8cd87016de351c4480b89d8ab63d26a0b2ff96cd347a2b25e5f8747181 WatchSource:0}: Error finding container cdebad8cd87016de351c4480b89d8ab63d26a0b2ff96cd347a2b25e5f8747181: Status 404 returned error can't find the container with id cdebad8cd87016de351c4480b89d8ab63d26a0b2ff96cd347a2b25e5f8747181
Jul 22 02:26:41.405450 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:41.405361    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" event={"ID":"8fea78ed-30b1-4b6a-a510-0924a507b5f9","Type":"ContainerStarted","Data":"d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752"}
Jul 22 02:26:41.405450 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:41.405417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" event={"ID":"8fea78ed-30b1-4b6a-a510-0924a507b5f9","Type":"ContainerStarted","Data":"cdebad8cd87016de351c4480b89d8ab63d26a0b2ff96cd347a2b25e5f8747181"}
Jul 22 02:26:41.424306 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:41.424258    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" podStartSLOduration=1.424244157 podStartE2EDuration="1.424244157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:26:41.422521235 +0000 UTC m=+2512.925379464" watchObservedRunningTime="2026-07-22 02:26:41.424244157 +0000 UTC m=+2512.927102385"
Jul 22 02:26:50.711084 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:50.711052    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:50.711084 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:50.711093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:50.723365 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:50.723341    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:26:51.464777 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:26:51.464745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:27:26.308834 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:26.308800    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"]
Jul 22 02:27:41.328320 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.328271    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" podUID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" containerName="main" containerID="cri-o://d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752" gracePeriod=45
Jul 22 02:27:41.454360 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.454328    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" podUID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 22 02:27:41.574507 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.574480    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:27:41.683296 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.683209    2561 generic.go:356] "Generic (PLEG): container finished" podID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" containerID="d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752" exitCode=0
Jul 22 02:27:41.683296 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.683249    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"
Jul 22 02:27:41.683296 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.683288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" event={"ID":"8fea78ed-30b1-4b6a-a510-0924a507b5f9","Type":"ContainerDied","Data":"d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752"}
Jul 22 02:27:41.683492 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.683326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq" event={"ID":"8fea78ed-30b1-4b6a-a510-0924a507b5f9","Type":"ContainerDied","Data":"cdebad8cd87016de351c4480b89d8ab63d26a0b2ff96cd347a2b25e5f8747181"}
Jul 22 02:27:41.683492 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.683344    2561 scope.go:117] "RemoveContainer" containerID="d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752"
Jul 22 02:27:41.689714 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.689682    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"]
Jul 22 02:27:41.690045 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.690033    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" containerName="main"
Jul 22 02:27:41.690096 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.690047    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" containerName="main"
Jul 22 02:27:41.690132 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.690102    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" containerName="main"
Jul 22 02:27:41.691862 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.691843    2561 scope.go:117] "RemoveContainer" containerID="d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752"
Jul 22 02:27:41.692115 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:27:41.692099    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752\": container with ID starting with d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752 not found: ID does not exist" containerID="d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752"
Jul 22 02:27:41.692165 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.692122    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752"} err="failed to get container status \"d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752\": rpc error: code = NotFound desc = could not find container \"d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752\": container with ID starting with d5f7ceba508b3cf6e86516d22d9ffa8836e3e831bbacc96a66e42a30eae12752 not found: ID does not exist"
Jul 22 02:27:41.693972 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.693955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.703493 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.703474    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"kube-root-ca.crt\""
Jul 22 02:27:41.703583 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.703474    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"openshift-service-ca.crt\""
Jul 22 02:27:41.703583 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.703474    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"default-dockercfg-8jfbh\""
Jul 22 02:27:41.703583 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.703474    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-000830\"/\"rate-test-model-000830-kserve-self-signed-certs\""
Jul 22 02:27:41.719853 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.719827    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"]
Jul 22 02:27:41.741659 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741634    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-model-cache\") pod \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") "
Jul 22 02:27:41.741802 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741699    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-dshm\") pod \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") "
Jul 22 02:27:41.741802 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t57kr\" (UniqueName: \"kubernetes.io/projected/8fea78ed-30b1-4b6a-a510-0924a507b5f9-kube-api-access-t57kr\") pod \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") "
Jul 22 02:27:41.742047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-home\") pod \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") "
Jul 22 02:27:41.742047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741894    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tmp-dir\") pod \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") "
Jul 22 02:27:41.742047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tls-certs\") pod \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\" (UID: \"8fea78ed-30b1-4b6a-a510-0924a507b5f9\") "
Jul 22 02:27:41.742047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.741922    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fea78ed-30b1-4b6a-a510-0924a507b5f9" (UID: "8fea78ed-30b1-4b6a-a510-0924a507b5f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:27:41.742261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.742080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-home" (OuterVolumeSpecName: "home") pod "8fea78ed-30b1-4b6a-a510-0924a507b5f9" (UID: "8fea78ed-30b1-4b6a-a510-0924a507b5f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:27:41.742261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.742094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fea78ed-30b1-4b6a-a510-0924a507b5f9" (UID: "8fea78ed-30b1-4b6a-a510-0924a507b5f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:27:41.742261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.742209    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:27:41.742261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.742231    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:27:41.742261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.742244    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:27:41.743802 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.743782    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-dshm" (OuterVolumeSpecName: "dshm") pod "8fea78ed-30b1-4b6a-a510-0924a507b5f9" (UID: "8fea78ed-30b1-4b6a-a510-0924a507b5f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:27:41.743953 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.743933    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fea78ed-30b1-4b6a-a510-0924a507b5f9" (UID: "8fea78ed-30b1-4b6a-a510-0924a507b5f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:27:41.744165 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.744148    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fea78ed-30b1-4b6a-a510-0924a507b5f9-kube-api-access-t57kr" (OuterVolumeSpecName: "kube-api-access-t57kr") pod "8fea78ed-30b1-4b6a-a510-0924a507b5f9" (UID: "8fea78ed-30b1-4b6a-a510-0924a507b5f9"). InnerVolumeSpecName "kube-api-access-t57kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:27:41.843309 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-tmp-dir\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.843500 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a3562-953d-4cd1-8e35-f5c584bc9b27-tls-certs\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.843500 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-dshm\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.843500 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-model-cache\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.843500 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-home\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.843681 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzqh\" (UniqueName: \"kubernetes.io/projected/327a3562-953d-4cd1-8e35-f5c584bc9b27-kube-api-access-hzzqh\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.843681 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843638    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fea78ed-30b1-4b6a-a510-0924a507b5f9-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:27:41.843681 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843666    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fea78ed-30b1-4b6a-a510-0924a507b5f9-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:27:41.843811 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.843680    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t57kr\" (UniqueName: \"kubernetes.io/projected/8fea78ed-30b1-4b6a-a510-0924a507b5f9-kube-api-access-t57kr\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:27:41.944957 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.944878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzqh\" (UniqueName: \"kubernetes.io/projected/327a3562-953d-4cd1-8e35-f5c584bc9b27-kube-api-access-hzzqh\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.944957 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.944922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-tmp-dir\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.944957 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.944945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a3562-953d-4cd1-8e35-f5c584bc9b27-tls-certs\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.945261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.945127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-dshm\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.945261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.945173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-model-cache\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.945261 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.945227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-home\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.945422 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.945385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-tmp-dir\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.945593 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.945574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-model-cache\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.945673 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.945594    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-home\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.947240 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.947221    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-dshm\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.947458 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.947440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a3562-953d-4cd1-8e35-f5c584bc9b27-tls-certs\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:41.973758 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:41.973717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzqh\" (UniqueName: \"kubernetes.io/projected/327a3562-953d-4cd1-8e35-f5c584bc9b27-kube-api-access-hzzqh\") pod \"rate-test-model-000830-kserve-6fdfdb48dc-2vbb2\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") " pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:42.003248 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.003221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:42.025125 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.025104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"]
Jul 22 02:27:42.031338 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.031315    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/sub-test-model-7eead4-kserve-6ccccdb88c-7z5mq"]
Jul 22 02:27:42.143837 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.143811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"]
Jul 22 02:27:42.145160 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:27:42.145135    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod327a3562_953d_4cd1_8e35_f5c584bc9b27.slice/crio-563e2f14c85ec07438a19f376e2f02064ffc8669fb973e5cc279ac56db3a3bff WatchSource:0}: Error finding container 563e2f14c85ec07438a19f376e2f02064ffc8669fb973e5cc279ac56db3a3bff: Status 404 returned error can't find the container with id 563e2f14c85ec07438a19f376e2f02064ffc8669fb973e5cc279ac56db3a3bff
Jul 22 02:27:42.693504 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.693465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2" event={"ID":"327a3562-953d-4cd1-8e35-f5c584bc9b27","Type":"ContainerStarted","Data":"673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c"}
Jul 22 02:27:42.694012 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.693520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2" event={"ID":"327a3562-953d-4cd1-8e35-f5c584bc9b27","Type":"ContainerStarted","Data":"563e2f14c85ec07438a19f376e2f02064ffc8669fb973e5cc279ac56db3a3bff"}
Jul 22 02:27:42.714530 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:42.714486    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2" podStartSLOduration=1.7144719149999998 podStartE2EDuration="1.714471915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:27:42.713020616 +0000 UTC m=+2574.215878855" watchObservedRunningTime="2026-07-22 02:27:42.714471915 +0000 UTC m=+2574.217330161"
Jul 22 02:27:43.102718 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:43.102675    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fea78ed-30b1-4b6a-a510-0924a507b5f9" path="/var/lib/kubelet/pods/8fea78ed-30b1-4b6a-a510-0924a507b5f9/volumes"
Jul 22 02:27:52.004194 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:52.004158    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:52.004194 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:52.004203    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:52.016502 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:52.016472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:27:52.205537 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:52.205504    2561 scope.go:117] "RemoveContainer" containerID="2a88bcdc6f87b36ddac621c072441963a5e97480e54c16ded1d92fd8388f7e7c"
Jul 22 02:27:52.755399 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:27:52.755372    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:28:05.484367 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.484332    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"]
Jul 22 02:28:05.487654 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.487637    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.498071 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.498049    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"default-dockercfg-h2zwx\""
Jul 22 02:28:05.498181 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.498101    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"openshift-service-ca.crt\""
Jul 22 02:28:05.502972 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.502956    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"kube-root-ca.crt\""
Jul 22 02:28:05.503081 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.503066    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7eead4\"/\"rate-test-model-7eead4-kserve-self-signed-certs\""
Jul 22 02:28:05.528285 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.528253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-home\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.528460 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.528318    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-dshm\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.528460 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.528380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tmp-dir\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.528460 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.528436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ptk\" (UniqueName: \"kubernetes.io/projected/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-kube-api-access-h9ptk\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.528620 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.528477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-model-cache\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.528620 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.528534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tls-certs\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.534706 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.534680    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"]
Jul 22 02:28:05.629679 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.629649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-dshm\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.629679 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.629680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tmp-dir\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.629933 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.629715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ptk\" (UniqueName: \"kubernetes.io/projected/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-kube-api-access-h9ptk\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.629933 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.629770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-model-cache\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.629933 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.629796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tls-certs\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.629933 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.629855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-home\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.630153 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.630067    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tmp-dir\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.630217 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.630178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-model-cache\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.630217 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.630195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-home\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.632029 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.632007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-dshm\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.632323 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.632303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tls-certs\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.641163 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.641136    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ptk\" (UniqueName: \"kubernetes.io/projected/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-kube-api-access-h9ptk\") pod \"rate-test-model-7eead4-kserve-b5fdc6875-9wp9c\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") " pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.797086 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.797057    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:05.985044 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:05.985016    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"]
Jul 22 02:28:05.987538 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:28:05.987512    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91a52f1_5b3d_4dc9_ad30_9dd994e0518a.slice/crio-e79fcbaef11152ee081801a892df00dc9a108c9fae3846f353b6573b4e7f2478 WatchSource:0}: Error finding container e79fcbaef11152ee081801a892df00dc9a108c9fae3846f353b6573b4e7f2478: Status 404 returned error can't find the container with id e79fcbaef11152ee081801a892df00dc9a108c9fae3846f353b6573b4e7f2478
Jul 22 02:28:06.822431 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:06.822391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c" event={"ID":"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a","Type":"ContainerStarted","Data":"dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061"}
Jul 22 02:28:06.822814 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:06.822442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c" event={"ID":"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a","Type":"ContainerStarted","Data":"e79fcbaef11152ee081801a892df00dc9a108c9fae3846f353b6573b4e7f2478"}
Jul 22 02:28:06.853372 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:06.853324    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c" podStartSLOduration=1.8533098890000002 podStartE2EDuration="1.853309889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:06.852140466 +0000 UTC m=+2598.354998695" watchObservedRunningTime="2026-07-22 02:28:06.853309889 +0000 UTC m=+2598.356168117"
Jul 22 02:28:15.797307 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:15.797273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:15.797718 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:15.797354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:15.810109 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:15.810076    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:15.877395 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:15.877369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:28:52.228038 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:52.228006    2561 scope.go:117] "RemoveContainer" containerID="f85ef4f7cb22f0e901650759a7c34b20c6283b3c0d55e6124af9df9f31d443ba"
Jul 22 02:28:52.236100 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:52.236072    2561 scope.go:117] "RemoveContainer" containerID="061e52b000eee3d1ba977ed29c806bead51274a456e6f0eab769d9696deff01d"
Jul 22 02:28:59.478845 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:28:59.478812    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"]
Jul 22 02:29:00.380513 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:00.380477    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"]
Jul 22 02:29:14.498415 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.498362    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2" podUID="327a3562-953d-4cd1-8e35-f5c584bc9b27" containerName="main" containerID="cri-o://673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c" gracePeriod=45
Jul 22 02:29:14.770021 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.769987    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:29:14.836995 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.836961    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzzqh\" (UniqueName: \"kubernetes.io/projected/327a3562-953d-4cd1-8e35-f5c584bc9b27-kube-api-access-hzzqh\") pod \"327a3562-953d-4cd1-8e35-f5c584bc9b27\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") "
Jul 22 02:29:14.837199 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-tmp-dir\") pod \"327a3562-953d-4cd1-8e35-f5c584bc9b27\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") "
Jul 22 02:29:14.837199 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-home\") pod \"327a3562-953d-4cd1-8e35-f5c584bc9b27\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") "
Jul 22 02:29:14.837199 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-model-cache\") pod \"327a3562-953d-4cd1-8e35-f5c584bc9b27\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") "
Jul 22 02:29:14.837199 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837150    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-dshm\") pod \"327a3562-953d-4cd1-8e35-f5c584bc9b27\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") "
Jul 22 02:29:14.837199 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a3562-953d-4cd1-8e35-f5c584bc9b27-tls-certs\") pod \"327a3562-953d-4cd1-8e35-f5c584bc9b27\" (UID: \"327a3562-953d-4cd1-8e35-f5c584bc9b27\") "
Jul 22 02:29:14.837475 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837293    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "327a3562-953d-4cd1-8e35-f5c584bc9b27" (UID: "327a3562-953d-4cd1-8e35-f5c584bc9b27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:14.837475 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-model-cache" (OuterVolumeSpecName: "model-cache") pod "327a3562-953d-4cd1-8e35-f5c584bc9b27" (UID: "327a3562-953d-4cd1-8e35-f5c584bc9b27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:14.837576 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837508    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:14.837722 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.837687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-home" (OuterVolumeSpecName: "home") pod "327a3562-953d-4cd1-8e35-f5c584bc9b27" (UID: "327a3562-953d-4cd1-8e35-f5c584bc9b27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:14.839282 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.839252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327a3562-953d-4cd1-8e35-f5c584bc9b27-kube-api-access-hzzqh" (OuterVolumeSpecName: "kube-api-access-hzzqh") pod "327a3562-953d-4cd1-8e35-f5c584bc9b27" (UID: "327a3562-953d-4cd1-8e35-f5c584bc9b27"). InnerVolumeSpecName "kube-api-access-hzzqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:14.839282 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.839269    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327a3562-953d-4cd1-8e35-f5c584bc9b27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "327a3562-953d-4cd1-8e35-f5c584bc9b27" (UID: "327a3562-953d-4cd1-8e35-f5c584bc9b27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:14.839533 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.839514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-dshm" (OuterVolumeSpecName: "dshm") pod "327a3562-953d-4cd1-8e35-f5c584bc9b27" (UID: "327a3562-953d-4cd1-8e35-f5c584bc9b27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:14.938051 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.938018    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzqh\" (UniqueName: \"kubernetes.io/projected/327a3562-953d-4cd1-8e35-f5c584bc9b27-kube-api-access-hzzqh\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:14.938051 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.938051    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:14.938219 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.938066    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:14.938219 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.938079    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/327a3562-953d-4cd1-8e35-f5c584bc9b27-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:14.938219 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:14.938091    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/327a3562-953d-4cd1-8e35-f5c584bc9b27-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:15.151358 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.151325    2561 generic.go:356] "Generic (PLEG): container finished" podID="327a3562-953d-4cd1-8e35-f5c584bc9b27" containerID="673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c" exitCode=0
Jul 22 02:29:15.151528 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.151366    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"
Jul 22 02:29:15.151528 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.151408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2" event={"ID":"327a3562-953d-4cd1-8e35-f5c584bc9b27","Type":"ContainerDied","Data":"673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c"}
Jul 22 02:29:15.151528 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.151452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2" event={"ID":"327a3562-953d-4cd1-8e35-f5c584bc9b27","Type":"ContainerDied","Data":"563e2f14c85ec07438a19f376e2f02064ffc8669fb973e5cc279ac56db3a3bff"}
Jul 22 02:29:15.151528 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.151475    2561 scope.go:117] "RemoveContainer" containerID="673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c"
Jul 22 02:29:15.159911 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.159847    2561 scope.go:117] "RemoveContainer" containerID="673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c"
Jul 22 02:29:15.160107 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:29:15.160091    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c\": container with ID starting with 673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c not found: ID does not exist" containerID="673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c"
Jul 22 02:29:15.160150 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.160113    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c"} err="failed to get container status \"673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c\": rpc error: code = NotFound desc = could not find container \"673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c\": container with ID starting with 673ba225730e1381a6d761275a89be23d4a241af68b4f192f1931ba778c0de4c not found: ID does not exist"
Jul 22 02:29:15.168058 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.168035    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"]
Jul 22 02:29:15.172319 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.172302    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-000830/rate-test-model-000830-kserve-6fdfdb48dc-2vbb2"]
Jul 22 02:29:15.400202 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.400108    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c" podUID="d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" containerName="main" containerID="cri-o://dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061" gracePeriod=45
Jul 22 02:29:15.650894 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.650841    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:29:15.743507 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-dshm\") pod \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") "
Jul 22 02:29:15.743705 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743523    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9ptk\" (UniqueName: \"kubernetes.io/projected/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-kube-api-access-h9ptk\") pod \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") "
Jul 22 02:29:15.743705 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743558    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tmp-dir\") pod \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") "
Jul 22 02:29:15.743705 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-model-cache\") pod \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") "
Jul 22 02:29:15.743705 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743623    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-home\") pod \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") "
Jul 22 02:29:15.743705 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tls-certs\") pod \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\" (UID: \"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a\") "
Jul 22 02:29:15.744023 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" (UID: "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:15.744023 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743874    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-home" (OuterVolumeSpecName: "home") pod "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" (UID: "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:15.744023 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.743902    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" (UID: "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:15.745693 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.745665    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" (UID: "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:15.745822 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.745669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-kube-api-access-h9ptk" (OuterVolumeSpecName: "kube-api-access-h9ptk") pod "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" (UID: "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a"). InnerVolumeSpecName "kube-api-access-h9ptk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:15.745822 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.745750    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-dshm" (OuterVolumeSpecName: "dshm") pod "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" (UID: "d91a52f1-5b3d-4dc9-ad30-9dd994e0518a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:29:15.844455 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.844424    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-home\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:15.844455 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.844454    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tls-certs\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:15.844663 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.844467    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-dshm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:15.844663 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.844479    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ptk\" (UniqueName: \"kubernetes.io/projected/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-kube-api-access-h9ptk\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:15.844663 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.844490    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-tmp-dir\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:15.844663 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:15.844502    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a-model-cache\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:16.158929 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.158900    2561 generic.go:356] "Generic (PLEG): container finished" podID="d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" containerID="dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061" exitCode=0
Jul 22 02:29:16.159102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.158941    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"
Jul 22 02:29:16.159102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.158971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c" event={"ID":"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a","Type":"ContainerDied","Data":"dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061"}
Jul 22 02:29:16.159102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.159003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c" event={"ID":"d91a52f1-5b3d-4dc9-ad30-9dd994e0518a","Type":"ContainerDied","Data":"e79fcbaef11152ee081801a892df00dc9a108c9fae3846f353b6573b4e7f2478"}
Jul 22 02:29:16.159102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.159019    2561 scope.go:117] "RemoveContainer" containerID="dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061"
Jul 22 02:29:16.168718 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.168699    2561 scope.go:117] "RemoveContainer" containerID="dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061"
Jul 22 02:29:16.168960 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:29:16.168941    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061\": container with ID starting with dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061 not found: ID does not exist" containerID="dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061"
Jul 22 02:29:16.169014 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.168969    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061"} err="failed to get container status \"dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061\": rpc error: code = NotFound desc = could not find container \"dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061\": container with ID starting with dba80f1533d4eee0653a47c3aa492c5662d1f88ab132b902a021d41a0bba9061 not found: ID does not exist"
Jul 22 02:29:16.180583 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.180561    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"]
Jul 22 02:29:16.184790 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:16.184769    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7eead4/rate-test-model-7eead4-kserve-b5fdc6875-9wp9c"]
Jul 22 02:29:17.102215 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:17.102180    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="327a3562-953d-4cd1-8e35-f5c584bc9b27" path="/var/lib/kubelet/pods/327a3562-953d-4cd1-8e35-f5c584bc9b27/volumes"
Jul 22 02:29:17.102668 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:17.102584    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" path="/var/lib/kubelet/pods/d91a52f1-5b3d-4dc9-ad30-9dd994e0518a/volumes"
Jul 22 02:29:19.204047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204015    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"]
Jul 22 02:29:19.204415 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204358    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" containerName="main"
Jul 22 02:29:19.204415 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204368    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" containerName="main"
Jul 22 02:29:19.204415 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204391    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="327a3562-953d-4cd1-8e35-f5c584bc9b27" containerName="main"
Jul 22 02:29:19.204415 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204397    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="327a3562-953d-4cd1-8e35-f5c584bc9b27" containerName="main"
Jul 22 02:29:19.204548 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204451    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="327a3562-953d-4cd1-8e35-f5c584bc9b27" containerName="main"
Jul 22 02:29:19.204548 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.204460    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91a52f1-5b3d-4dc9-ad30-9dd994e0518a" containerName="main"
Jul 22 02:29:19.208621 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.208604    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:19.211108 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.211080    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:29:19.215508 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.215473    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"]
Jul 22 02:29:19.273774 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.273742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-000830-7xcgt\" (UID: \"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:19.374641 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.374603    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-000830-7xcgt\" (UID: \"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:19.375260 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.375239    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-000830-7xcgt\" (UID: \"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:19.520815 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.520780    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:19.654084 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:19.654049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"]
Jul 22 02:29:19.657415 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:29:19.657389    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6b6066_c1f4_4c01_bd7c_2c7c9470d8fb.slice/crio-9dd61b22660e0ca4cb58d9d4e53ded545ec2dbf99fd2b172bf314797f001e71a WatchSource:0}: Error finding container 9dd61b22660e0ca4cb58d9d4e53ded545ec2dbf99fd2b172bf314797f001e71a: Status 404 returned error can't find the container with id 9dd61b22660e0ca4cb58d9d4e53ded545ec2dbf99fd2b172bf314797f001e71a
Jul 22 02:29:20.185184 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:20.185100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt" event={"ID":"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb","Type":"ContainerStarted","Data":"9a187ad3fbf17c5ef8dce53550f647506cab0baa59818c02f2984260affa1a0f"}
Jul 22 02:29:20.185184 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:20.185153    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt" event={"ID":"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb","Type":"ContainerStarted","Data":"9dd61b22660e0ca4cb58d9d4e53ded545ec2dbf99fd2b172bf314797f001e71a"}
Jul 22 02:29:20.200948 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:20.200903    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt" podStartSLOduration=1.200888219 podStartE2EDuration="1.200888219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:20.200162773 +0000 UTC m=+2671.703021025" watchObservedRunningTime="2026-07-22 02:29:20.200888219 +0000 UTC m=+2671.703746447"
Jul 22 02:29:21.191233 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:21.191198    2561 generic.go:356] "Generic (PLEG): container finished" podID="9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb" containerID="9a187ad3fbf17c5ef8dce53550f647506cab0baa59818c02f2984260affa1a0f" exitCode=0
Jul 22 02:29:21.191638 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:21.191295    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt" event={"ID":"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb","Type":"ContainerDied","Data":"9a187ad3fbf17c5ef8dce53550f647506cab0baa59818c02f2984260affa1a0f"}
Jul 22 02:29:22.327998 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:22.327976    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:22.399648 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:22.399615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb-maas-api-service-ca\") pod \"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb\" (UID: \"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb\") "
Jul 22 02:29:22.400018 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:22.399992    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb" (UID: "9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:29:22.501251 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:22.501175    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:23.201300 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:23.201272    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"
Jul 22 02:29:23.201519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:23.201278    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt" event={"ID":"9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb","Type":"ContainerDied","Data":"9dd61b22660e0ca4cb58d9d4e53ded545ec2dbf99fd2b172bf314797f001e71a"}
Jul 22 02:29:23.201519 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:23.201382    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dd61b22660e0ca4cb58d9d4e53ded545ec2dbf99fd2b172bf314797f001e71a"
Jul 22 02:29:29.687020 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:29.686985    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"]
Jul 22 02:29:29.687465 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:29.687231    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5" podUID="869caafd-43de-44b2-acd1-285902050748" containerName="maas-api" containerID="cri-o://88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82" gracePeriod=30
Jul 22 02:29:29.928835 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:29.928811    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:29:30.065041 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.065007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls\") pod \"869caafd-43de-44b2-acd1-285902050748\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") "
Jul 22 02:29:30.065041 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.065041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/869caafd-43de-44b2-acd1-285902050748-openshift-service-ca\") pod \"869caafd-43de-44b2-acd1-285902050748\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") "
Jul 22 02:29:30.065236 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.065072    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crnxr\" (UniqueName: \"kubernetes.io/projected/869caafd-43de-44b2-acd1-285902050748-kube-api-access-crnxr\") pod \"869caafd-43de-44b2-acd1-285902050748\" (UID: \"869caafd-43de-44b2-acd1-285902050748\") "
Jul 22 02:29:30.065401 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.065377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869caafd-43de-44b2-acd1-285902050748-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "869caafd-43de-44b2-acd1-285902050748" (UID: "869caafd-43de-44b2-acd1-285902050748"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:29:30.067226 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.067205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869caafd-43de-44b2-acd1-285902050748-kube-api-access-crnxr" (OuterVolumeSpecName: "kube-api-access-crnxr") pod "869caafd-43de-44b2-acd1-285902050748" (UID: "869caafd-43de-44b2-acd1-285902050748"). InnerVolumeSpecName "kube-api-access-crnxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:30.067377 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.067356    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "869caafd-43de-44b2-acd1-285902050748" (UID: "869caafd-43de-44b2-acd1-285902050748"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:30.166355 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.166320    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/869caafd-43de-44b2-acd1-285902050748-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:30.166355 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.166353    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/869caafd-43de-44b2-acd1-285902050748-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:30.166547 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.166369    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crnxr\" (UniqueName: \"kubernetes.io/projected/869caafd-43de-44b2-acd1-285902050748-kube-api-access-crnxr\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:30.236823 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.236777    2561 generic.go:356] "Generic (PLEG): container finished" podID="869caafd-43de-44b2-acd1-285902050748" containerID="88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82" exitCode=0
Jul 22 02:29:30.236974 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.236912    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"
Jul 22 02:29:30.236974 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.236916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5" event={"ID":"869caafd-43de-44b2-acd1-285902050748","Type":"ContainerDied","Data":"88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82"}
Jul 22 02:29:30.237098 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.236997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5" event={"ID":"869caafd-43de-44b2-acd1-285902050748","Type":"ContainerDied","Data":"90cd8a2d5b5b158908ec35b8a0f889e97707d7fe887020a13c503446cac028c5"}
Jul 22 02:29:30.237098 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.237028    2561 scope.go:117] "RemoveContainer" containerID="88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82"
Jul 22 02:29:30.246117 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.246103    2561 scope.go:117] "RemoveContainer" containerID="88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82"
Jul 22 02:29:30.246404 ip-10-0-130-189 kubenswrapper[2561]: E0722 02:29:30.246382    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82\": container with ID starting with 88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82 not found: ID does not exist" containerID="88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82"
Jul 22 02:29:30.246498 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.246412    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82"} err="failed to get container status \"88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82\": rpc error: code = NotFound desc = could not find container \"88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82\": container with ID starting with 88091b79cd716310123556dc507a391e737697c2a64a372fb688149fc77e7f82 not found: ID does not exist"
Jul 22 02:29:30.260910 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.260887    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"]
Jul 22 02:29:30.264461 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:30.264443    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-000830-8454c77556-cm9f5"]
Jul 22 02:29:31.102299 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:31.102261    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="869caafd-43de-44b2-acd1-285902050748" path="/var/lib/kubelet/pods/869caafd-43de-44b2-acd1-285902050748/volumes"
Jul 22 02:29:34.297091 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:34.297061    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"]
Jul 22 02:29:34.298656 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:34.298633    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-000830-7xcgt"]
Jul 22 02:29:35.101634 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:35.101602    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb" path="/var/lib/kubelet/pods/9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb/volumes"
Jul 22 02:29:41.666910 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.666833    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"]
Jul 22 02:29:41.667273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.667159    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb" containerName="revoke-keys"
Jul 22 02:29:41.667273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.667170    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb" containerName="revoke-keys"
Jul 22 02:29:41.667273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.667188    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869caafd-43de-44b2-acd1-285902050748" containerName="maas-api"
Jul 22 02:29:41.667273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.667194    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="869caafd-43de-44b2-acd1-285902050748" containerName="maas-api"
Jul 22 02:29:41.667273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.667249    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="869caafd-43de-44b2-acd1-285902050748" containerName="maas-api"
Jul 22 02:29:41.667273 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.667256    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f6b6066-c1f4-4c01-bd7c-2c7c9470d8fb" containerName="revoke-keys"
Jul 22 02:29:41.671300 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.671282    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:41.673776 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.673749    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:29:41.678102 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.678076    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"]
Jul 22 02:29:41.763229 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.763193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c88c3-a09c-4ee1-adb0-8da416dc8695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw\" (UID: \"149c88c3-a09c-4ee1-adb0-8da416dc8695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:41.864120 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.864086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c88c3-a09c-4ee1-adb0-8da416dc8695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw\" (UID: \"149c88c3-a09c-4ee1-adb0-8da416dc8695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:41.864690 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.864671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c88c3-a09c-4ee1-adb0-8da416dc8695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw\" (UID: \"149c88c3-a09c-4ee1-adb0-8da416dc8695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:41.983202 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:41.983133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:42.115208 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:42.115184    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"]
Jul 22 02:29:42.116744 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:29:42.116697    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149c88c3_a09c_4ee1_adb0_8da416dc8695.slice/crio-2b394495d862054a5f146fdfd465cd4f77fc0183c89266cb16d8c2cd006f4109 WatchSource:0}: Error finding container 2b394495d862054a5f146fdfd465cd4f77fc0183c89266cb16d8c2cd006f4109: Status 404 returned error can't find the container with id 2b394495d862054a5f146fdfd465cd4f77fc0183c89266cb16d8c2cd006f4109
Jul 22 02:29:42.295587 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:42.295560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw" event={"ID":"149c88c3-a09c-4ee1-adb0-8da416dc8695","Type":"ContainerStarted","Data":"2b394495d862054a5f146fdfd465cd4f77fc0183c89266cb16d8c2cd006f4109"}
Jul 22 02:29:43.301028 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:43.300992    2561 generic.go:356] "Generic (PLEG): container finished" podID="149c88c3-a09c-4ee1-adb0-8da416dc8695" containerID="db7298c8390a06853fda0350311241e8e0e963854ec50ae196249b7ac90c7b8b" exitCode=0
Jul 22 02:29:43.301411 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:43.301078    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw" event={"ID":"149c88c3-a09c-4ee1-adb0-8da416dc8695","Type":"ContainerDied","Data":"db7298c8390a06853fda0350311241e8e0e963854ec50ae196249b7ac90c7b8b"}
Jul 22 02:29:44.427696 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:44.427674    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:44.584753 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:44.584659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c88c3-a09c-4ee1-adb0-8da416dc8695-maas-api-service-ca\") pod \"149c88c3-a09c-4ee1-adb0-8da416dc8695\" (UID: \"149c88c3-a09c-4ee1-adb0-8da416dc8695\") "
Jul 22 02:29:44.585023 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:44.584999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149c88c3-a09c-4ee1-adb0-8da416dc8695-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "149c88c3-a09c-4ee1-adb0-8da416dc8695" (UID: "149c88c3-a09c-4ee1-adb0-8da416dc8695"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:29:44.686157 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:44.686127    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149c88c3-a09c-4ee1-adb0-8da416dc8695-maas-api-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:45.311019 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:45.310985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw" event={"ID":"149c88c3-a09c-4ee1-adb0-8da416dc8695","Type":"ContainerDied","Data":"2b394495d862054a5f146fdfd465cd4f77fc0183c89266cb16d8c2cd006f4109"}
Jul 22 02:29:45.311019 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:45.311008    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"
Jul 22 02:29:45.311211 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:45.311037    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b394495d862054a5f146fdfd465cd4f77fc0183c89266cb16d8c2cd006f4109"
Jul 22 02:29:49.181816 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:49.181790    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:29:49.184397 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:49.184375    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf44z_4eeb6b09-899e-4336-8693-99407d362af3/ovn-acl-logging/0.log"
Jul 22 02:29:52.112011 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.111981    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"]
Jul 22 02:29:52.112412 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.112215    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" podUID="c810c3de-71b4-4783-adc6-7cc13f273b4c" containerName="maas-api" containerID="cri-o://fe81b7eccfb48f5c093c4567a982a5d93fa287c404ac304aed60d45bd58a01f6" gracePeriod=30
Jul 22 02:29:52.345070 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.345038    2561 generic.go:356] "Generic (PLEG): container finished" podID="c810c3de-71b4-4783-adc6-7cc13f273b4c" containerID="fe81b7eccfb48f5c093c4567a982a5d93fa287c404ac304aed60d45bd58a01f6" exitCode=0
Jul 22 02:29:52.345192 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.345102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" event={"ID":"c810c3de-71b4-4783-adc6-7cc13f273b4c","Type":"ContainerDied","Data":"fe81b7eccfb48f5c093c4567a982a5d93fa287c404ac304aed60d45bd58a01f6"}
Jul 22 02:29:52.364142 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.364088    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:29:52.448363 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.448327    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls\") pod \"c810c3de-71b4-4783-adc6-7cc13f273b4c\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") "
Jul 22 02:29:52.448545 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.448377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810c3de-71b4-4783-adc6-7cc13f273b4c-openshift-service-ca\") pod \"c810c3de-71b4-4783-adc6-7cc13f273b4c\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") "
Jul 22 02:29:52.448545 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.448460    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2pkm\" (UniqueName: \"kubernetes.io/projected/c810c3de-71b4-4783-adc6-7cc13f273b4c-kube-api-access-v2pkm\") pod \"c810c3de-71b4-4783-adc6-7cc13f273b4c\" (UID: \"c810c3de-71b4-4783-adc6-7cc13f273b4c\") "
Jul 22 02:29:52.448822 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.448797    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c810c3de-71b4-4783-adc6-7cc13f273b4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c810c3de-71b4-4783-adc6-7cc13f273b4c" (UID: "c810c3de-71b4-4783-adc6-7cc13f273b4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:29:52.450356 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.450328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c810c3de-71b4-4783-adc6-7cc13f273b4c" (UID: "c810c3de-71b4-4783-adc6-7cc13f273b4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:29:52.450492 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.450475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c810c3de-71b4-4783-adc6-7cc13f273b4c-kube-api-access-v2pkm" (OuterVolumeSpecName: "kube-api-access-v2pkm") pod "c810c3de-71b4-4783-adc6-7cc13f273b4c" (UID: "c810c3de-71b4-4783-adc6-7cc13f273b4c"). InnerVolumeSpecName "kube-api-access-v2pkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:29:52.550045 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.550009    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pkm\" (UniqueName: \"kubernetes.io/projected/c810c3de-71b4-4783-adc6-7cc13f273b4c-kube-api-access-v2pkm\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:52.550045 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.550038    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c810c3de-71b4-4783-adc6-7cc13f273b4c-maas-api-tls\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:52.550045 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:52.550048    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810c3de-71b4-4783-adc6-7cc13f273b4c-openshift-service-ca\") on node \"ip-10-0-130-189.ec2.internal\" DevicePath \"\""
Jul 22 02:29:53.350983 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:53.350950    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"
Jul 22 02:29:53.350983 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:53.350960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm" event={"ID":"c810c3de-71b4-4783-adc6-7cc13f273b4c","Type":"ContainerDied","Data":"009ce7c685edf762f2fc3c579147f316d31ccfb6a7e4d441eaad975a54987b05"}
Jul 22 02:29:53.351502 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:53.351028    2561 scope.go:117] "RemoveContainer" containerID="fe81b7eccfb48f5c093c4567a982a5d93fa287c404ac304aed60d45bd58a01f6"
Jul 22 02:29:53.368678 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:53.368649    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"]
Jul 22 02:29:53.374382 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:53.374360    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-7eead4-697bd6fbb-5fnwm"]
Jul 22 02:29:55.102489 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:55.102455    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c810c3de-71b4-4783-adc6-7cc13f273b4c" path="/var/lib/kubelet/pods/c810c3de-71b4-4783-adc6-7cc13f273b4c/volumes"
Jul 22 02:29:56.777449 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:56.777415    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"]
Jul 22 02:29:56.782713 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:56.782685    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-7eead4-zjdfw"]
Jul 22 02:29:57.101822 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:29:57.101748    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="149c88c3-a09c-4ee1-adb0-8da416dc8695" path="/var/lib/kubelet/pods/149c88c3-a09c-4ee1-adb0-8da416dc8695/volumes"
Jul 22 02:30:00.138257 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138221    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"]
Jul 22 02:30:00.138832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138571    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149c88c3-a09c-4ee1-adb0-8da416dc8695" containerName="revoke-keys"
Jul 22 02:30:00.138832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="149c88c3-a09c-4ee1-adb0-8da416dc8695" containerName="revoke-keys"
Jul 22 02:30:00.138832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138594    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c810c3de-71b4-4783-adc6-7cc13f273b4c" containerName="maas-api"
Jul 22 02:30:00.138832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138600    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c810c3de-71b4-4783-adc6-7cc13f273b4c" containerName="maas-api"
Jul 22 02:30:00.138832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138665    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="149c88c3-a09c-4ee1-adb0-8da416dc8695" containerName="revoke-keys"
Jul 22 02:30:00.138832 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.138674    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c810c3de-71b4-4783-adc6-7cc13f273b4c" containerName="maas-api"
Jul 22 02:30:00.141627 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.141607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"
Jul 22 02:30:00.144027 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.144002    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qmc8m\""
Jul 22 02:30:00.150170 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.150140    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"]
Jul 22 02:30:00.152543 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.152524    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"
Jul 22 02:30:00.289095 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.289066    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"]
Jul 22 02:30:00.291783 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:30:00.291756    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe3cde0_a1bf_4336_93ff_e166be68e580.slice/crio-35023b0debb09a2b17f76db8da34e08f5163fca00b0fe610eafa750c1d922bcf WatchSource:0}: Error finding container 35023b0debb09a2b17f76db8da34e08f5163fca00b0fe610eafa750c1d922bcf: Status 404 returned error can't find the container with id 35023b0debb09a2b17f76db8da34e08f5163fca00b0fe610eafa750c1d922bcf
Jul 22 02:30:00.387015 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:00.386981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b" event={"ID":"5fe3cde0-a1bf-4336-93ff-e166be68e580","Type":"ContainerStarted","Data":"35023b0debb09a2b17f76db8da34e08f5163fca00b0fe610eafa750c1d922bcf"}
Jul 22 02:30:01.393101 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:01.393063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b" event={"ID":"5fe3cde0-a1bf-4336-93ff-e166be68e580","Type":"ContainerStarted","Data":"81f8fec88221a6e90bec77fc28c82f2d3e27f9ba1b8e67d4486c654e37afeebb"}
Jul 22 02:30:01.407663 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:01.407609    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b" podStartSLOduration=1.4075936709999999 podStartE2EDuration="1.407593671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:01.406995068 +0000 UTC m=+2712.909853297" watchObservedRunningTime="2026-07-22 02:30:01.407593671 +0000 UTC m=+2712.910451909"
Jul 22 02:30:02.399058 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:02.399019    2561 generic.go:356] "Generic (PLEG): container finished" podID="5fe3cde0-a1bf-4336-93ff-e166be68e580" containerID="81f8fec88221a6e90bec77fc28c82f2d3e27f9ba1b8e67d4486c654e37afeebb" exitCode=0
Jul 22 02:30:02.399437 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:02.399109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b" event={"ID":"5fe3cde0-a1bf-4336-93ff-e166be68e580","Type":"ContainerDied","Data":"81f8fec88221a6e90bec77fc28c82f2d3e27f9ba1b8e67d4486c654e37afeebb"}
Jul 22 02:30:03.543535 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:03.543508    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"
Jul 22 02:30:04.411156 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:04.411128    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b"
Jul 22 02:30:04.411327 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:04.411133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744790-md92b" event={"ID":"5fe3cde0-a1bf-4336-93ff-e166be68e580","Type":"ContainerDied","Data":"35023b0debb09a2b17f76db8da34e08f5163fca00b0fe610eafa750c1d922bcf"}
Jul 22 02:30:04.411327 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:04.411260    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35023b0debb09a2b17f76db8da34e08f5163fca00b0fe610eafa750c1d922bcf"
Jul 22 02:30:09.665444 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:09.665417    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nf8ht_2984e018-c6fe-45ee-b3c9-e32cd38b2841/manager/0.log"
Jul 22 02:30:09.798035 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:09.798008    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4dk5f_8f01f09c-61ed-4df8-aa8c-499ec8785950/manager/0.log"
Jul 22 02:30:10.052491 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:10.052456    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ns8d9_4fb02595-7f25-4b63-aabc-d0a01f48cd40/server/0.log"
Jul 22 02:30:10.304913 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:10.304837    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-gdm4q_d7138d62-ba39-428d-b2d6-845706592431/copy-manifests/0.log"
Jul 22 02:30:10.312205 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:10.312176    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-gdm4q_d7138d62-ba39-428d-b2d6-845706592431/manager/0.log"
Jul 22 02:30:11.204938 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:11.204905    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744775-xxn5w_16a4dc39-3572-4dcf-a4f3-d46c0ba4dccb/cleanup/0.log"
Jul 22 02:30:11.323689 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:11.323665    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744790-md92b_5fe3cde0-a1bf-4336-93ff-e166be68e580/cleanup/0.log"
Jul 22 02:30:11.456321 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:11.456231    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-7j62c_2cd6b3e8-ed8b-4d34-9247-b07dad382d1f/postgres/0.log"
Jul 22 02:30:12.978267 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:12.978239    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-j484c_f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e/manager/0.log"
Jul 22 02:30:13.115973 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:13.115945    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nf65f_a45c1928-0e71-42c8-be8a-c3219dba4c4e/manager/0.log"
Jul 22 02:30:13.260133 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:13.260107    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7tsfk_af95db8b-b50f-40ab-b971-a779024a53d0/kuadrant-console-plugin/0.log"
Jul 22 02:30:13.394303 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:13.394273    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tn7qs_73da48b7-8657-4665-b5c7-8615a13e790e/registry-server/0.log"
Jul 22 02:30:13.529886 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:13.529794    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rs68l_5fce6dbc-c000-4506-a27a-bbcd4d32cec3/manager/0.log"
Jul 22 02:30:13.815775 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:13.815679    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-wsdgd_8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c/manager/0.log"
Jul 22 02:30:14.596294 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:14.596255    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55d847747c-xdp4r_f9d366fd-d3b6-4c08-a513-296e9b2f991e/kube-auth-proxy/0.log"
Jul 22 02:30:14.855479 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:14.855388    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-cmc7n_ef78394a-1093-4b11-9be4-2197125d314c/payload-pre-processing/0.log"
Jul 22 02:30:15.492859 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:15.492831    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qsz4j_0ace246d-5612-42b5-af14-37c2f9af6329/main/0.log"
Jul 22 02:30:15.978384 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:15.978345    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-kv6gc_438bea22-072e-49b6-9957-97b4bd5f83c6/main/0.log"
Jul 22 02:30:16.102852 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:16.102823    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5fzkz_624f147e-05a1-459e-99ee-47d45e56c1ff/main/0.log"
Jul 22 02:30:24.717252 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:24.717223    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sqzb4_e297bc0c-22b3-4e81-a4b4-ae416d6ae24e/global-pull-secret-syncer/0.log"
Jul 22 02:30:24.870449 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:24.870423    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m4hg5_f0f19b6e-6eec-4ad8-af2f-69c5844b7abd/konnectivity-agent/0.log"
Jul 22 02:30:24.948461 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:24.948435    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-189.ec2.internal_1f6a1ecb72d07fd09a4dca395487a6f2/haproxy/0.log"
Jul 22 02:30:30.214964 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:30.214935    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-j484c_f9f2bb8c-16eb-4bcf-9e01-8536ad6a803e/manager/0.log"
Jul 22 02:30:30.312785 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:30.312753    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nf65f_a45c1928-0e71-42c8-be8a-c3219dba4c4e/manager/0.log"
Jul 22 02:30:30.384997 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:30.384972    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7tsfk_af95db8b-b50f-40ab-b971-a779024a53d0/kuadrant-console-plugin/0.log"
Jul 22 02:30:30.494181 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:30.494094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tn7qs_73da48b7-8657-4665-b5c7-8615a13e790e/registry-server/0.log"
Jul 22 02:30:30.763464 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:30.763429    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-rs68l_5fce6dbc-c000-4506-a27a-bbcd4d32cec3/manager/0.log"
Jul 22 02:30:30.912405 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:30.912370    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-wsdgd_8b3357b1-4e3b-4d8f-a03f-d2e3c4c61f9c/manager/0.log"
Jul 22 02:30:33.256573 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:33.256546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97gct_d7256c0d-e840-45aa-9964-46cd09750f78/node-exporter/0.log"
Jul 22 02:30:33.300017 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:33.299988    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97gct_d7256c0d-e840-45aa-9964-46cd09750f78/kube-rbac-proxy/0.log"
Jul 22 02:30:33.339548 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:33.339524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-97gct_d7256c0d-e840-45aa-9964-46cd09750f78/init-textfile/0.log"
Jul 22 02:30:34.451670 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.451641    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"]
Jul 22 02:30:34.452057 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.452005    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe3cde0-a1bf-4336-93ff-e166be68e580" containerName="cleanup"
Jul 22 02:30:34.452057 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.452016    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe3cde0-a1bf-4336-93ff-e166be68e580" containerName="cleanup"
Jul 22 02:30:34.452129 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.452074    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe3cde0-a1bf-4336-93ff-e166be68e580" containerName="cleanup"
Jul 22 02:30:34.455076 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.455060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.461679 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.461574    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6wcsf\"/\"openshift-service-ca.crt\""
Jul 22 02:30:34.462611 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.462596    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6wcsf\"/\"default-dockercfg-j2hsx\""
Jul 22 02:30:34.462691 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.462639    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6wcsf\"/\"kube-root-ca.crt\""
Jul 22 02:30:34.471140 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.471118    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"]
Jul 22 02:30:34.613584 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.613545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-lib-modules\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.613584 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.613585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-sys\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.613850 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.613643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hqc\" (UniqueName: \"kubernetes.io/projected/bde93be3-b11e-4b5a-a858-7a900e0f2817-kube-api-access-s6hqc\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.613850 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.613701    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-proc\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.613850 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.613766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-podres\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714676 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hqc\" (UniqueName: \"kubernetes.io/projected/bde93be3-b11e-4b5a-a858-7a900e0f2817-kube-api-access-s6hqc\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714676 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-proc\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714676 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714666    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-podres\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-lib-modules\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-sys\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-proc\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-podres\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-sys\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.714986 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.714907    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bde93be3-b11e-4b5a-a858-7a900e0f2817-lib-modules\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.725047 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.725019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hqc\" (UniqueName: \"kubernetes.io/projected/bde93be3-b11e-4b5a-a858-7a900e0f2817-kube-api-access-s6hqc\") pod \"perf-node-gather-daemonset-jbx7r\" (UID: \"bde93be3-b11e-4b5a-a858-7a900e0f2817\") " pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.764903 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.764871    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:34.907081 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:34.907054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"]
Jul 22 02:30:34.907398 ip-10-0-130-189 kubenswrapper[2561]: W0722 02:30:34.907376    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbde93be3_b11e_4b5a_a858_7a900e0f2817.slice/crio-3f9f5810aaa37391328a42131ee63695fed2bc07bbee10f98d913af95480643f WatchSource:0}: Error finding container 3f9f5810aaa37391328a42131ee63695fed2bc07bbee10f98d913af95480643f: Status 404 returned error can't find the container with id 3f9f5810aaa37391328a42131ee63695fed2bc07bbee10f98d913af95480643f
Jul 22 02:30:35.556122 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:35.556085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r" event={"ID":"bde93be3-b11e-4b5a-a858-7a900e0f2817","Type":"ContainerStarted","Data":"abf93342cfd7ea8aab99c6b0860b9396af5fe36805c471cb907e091d77c267ca"}
Jul 22 02:30:35.556510 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:35.556139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r" event={"ID":"bde93be3-b11e-4b5a-a858-7a900e0f2817","Type":"ContainerStarted","Data":"3f9f5810aaa37391328a42131ee63695fed2bc07bbee10f98d913af95480643f"}
Jul 22 02:30:35.556510 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:35.556185    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:35.578674 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:35.578625    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r" podStartSLOduration=1.578609561 podStartE2EDuration="1.578609561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:35.576943024 +0000 UTC m=+2747.079801254" watchObservedRunningTime="2026-07-22 02:30:35.578609561 +0000 UTC m=+2747.081467790"
Jul 22 02:30:38.701024 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:38.700998    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jwnlg_52e130bf-16ac-457b-bff4-e49e11b0e3bb/dns/0.log"
Jul 22 02:30:38.727407 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:38.727381    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jwnlg_52e130bf-16ac-457b-bff4-e49e11b0e3bb/kube-rbac-proxy/0.log"
Jul 22 02:30:38.848580 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:38.848543    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t8j9b_b2a4e8df-2d58-4fa6-a084-484802fabbbf/dns-node-resolver/0.log"
Jul 22 02:30:39.486873 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:39.486846    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-66595658cd-zvfqd_3950bd94-18cd-4904-a452-2b3d4e307da3/registry/0.log"
Jul 22 02:30:39.565380 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:39.565348    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s77x5_cab3a42b-231d-4f4e-bbec-1ed8e9421ecc/node-ca/0.log"
Jul 22 02:30:40.945530 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:40.945501    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55d847747c-xdp4r_f9d366fd-d3b6-4c08-a513-296e9b2f991e/kube-auth-proxy/0.log"
Jul 22 02:30:41.077541 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:41.077511    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-cmc7n_ef78394a-1093-4b11-9be4-2197125d314c/payload-pre-processing/0.log"
Jul 22 02:30:41.569794 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:41.569768    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6wcsf/perf-node-gather-daemonset-jbx7r"
Jul 22 02:30:42.102340 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:42.102306    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z74wb_30f6828a-7509-4862-b361-7946baf84f45/serve-healthcheck-canary/0.log"
Jul 22 02:30:42.694805 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:42.694757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7zvh_32a06c56-9fab-49ba-88ce-6606585a4e0f/kube-rbac-proxy/0.log"
Jul 22 02:30:42.720873 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:42.720844    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7zvh_32a06c56-9fab-49ba-88ce-6606585a4e0f/exporter/0.log"
Jul 22 02:30:42.746563 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:42.746540    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d7zvh_32a06c56-9fab-49ba-88ce-6606585a4e0f/extractor/0.log"
Jul 22 02:30:45.152054 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:45.152021    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55d847747c-xdp4r_f9d366fd-d3b6-4c08-a513-296e9b2f991e/kube-auth-proxy/0.log"
Jul 22 02:30:45.248907 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:45.248873    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-cmc7n_ef78394a-1093-4b11-9be4-2197125d314c/payload-pre-processing/0.log"
Jul 22 02:30:45.451711 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:45.451684    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nf8ht_2984e018-c6fe-45ee-b3c9-e32cd38b2841/manager/0.log"
Jul 22 02:30:45.780549 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:45.780518    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4dk5f_8f01f09c-61ed-4df8-aa8c-499ec8785950/manager/0.log"
Jul 22 02:30:46.001546 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:46.001518    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ns8d9_4fb02595-7f25-4b63-aabc-d0a01f48cd40/server/0.log"
Jul 22 02:30:46.156629 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:46.156601    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-gdm4q_d7138d62-ba39-428d-b2d6-845706592431/manager/0.log"
Jul 22 02:30:46.180440 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:46.180411    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585784b644-gdm4q_d7138d62-ba39-428d-b2d6-845706592431/copy-manifests/0.log"
Jul 22 02:30:46.689184 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:46.689155    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55d847747c-xdp4r_f9d366fd-d3b6-4c08-a513-296e9b2f991e/kube-auth-proxy/0.log"
Jul 22 02:30:46.799889 ip-10-0-130-189 kubenswrapper[2561]: I0722 02:30:46.799854    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-cmc7n_ef78394a-1093-4b11-9be4-2197125d314c/payload-pre-processing/0.log"