Jul 08 23:07:30.903987 ip-10-0-130-115 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 23:07:31.249308 ip-10-0-130-115 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 23:07:31.249308 ip-10-0-130-115 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 23:07:31.249308 ip-10-0-130-115 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 23:07:31.249308 ip-10-0-130-115 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 23:07:31.249308 ip-10-0-130-115 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 23:07:31.250133 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.249978    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 23:07:31.256783 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256769    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 23:07:31.256783 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256783    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256787    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256791    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256801    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256804    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256808    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256810    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256813    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256816    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256819    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256821    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256824    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256826    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256829    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256832    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256835    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256837    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256840    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256842    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 23:07:31.256845 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256844    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256847    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256850    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256853    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256856    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256858    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256861    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256864    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256867    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256869    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256872    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256874    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256877    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256879    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256882    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256884    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256887    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256889    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256892    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256894    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 23:07:31.257282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256898    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256902    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256906    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256908    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256911    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256913    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256916    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256919    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256922    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256924    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256927    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256930    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256935    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256939    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256942    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256945    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256948    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256950    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256953    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 23:07:31.257779 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256956    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256958    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256961    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256963    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256966    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256968    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256971    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256973    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256976    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256978    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256981    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256985    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256988    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256991    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256994    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.256997    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257000    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257002    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257005    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257007    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 23:07:31.258279 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257010    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257013    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257015    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257017    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257020    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257022    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257026    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257029    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257032    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257034    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257037    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257039    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257042    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257044    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257047    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257049    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257051    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257054    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257056    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257059    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 23:07:31.258769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257749    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257755    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257759    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257762    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257766    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257768    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257771    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257774    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257776    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257779    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257782    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257784    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257787    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257790    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257793    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257795    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257798    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257800    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257804    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257807    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 23:07:31.259238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257810    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257813    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257816    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257818    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257821    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257824    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257826    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257829    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257831    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257834    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257838    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257842    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257845    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257848    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257851    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257854    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257857    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257861    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 23:07:31.259769 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257865    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257868    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257871    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257873    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257876    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257879    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257881    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257884    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257887    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257889    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257892    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257894    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257899    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257901    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257904    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257906    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257909    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257912    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257914    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257917    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 23:07:31.260238 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257919    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257922    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257924    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257928    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257932    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257934    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257937    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257940    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257943    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257945    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257947    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257950    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257953    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257956    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257958    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257961    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257963    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257966    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257968    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257971    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 23:07:31.260720 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257973    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257976    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257978    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257981    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257984    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257987    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257990    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257992    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257995    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.257997    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258000    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258002    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258005    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258008    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258010    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258012    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258015    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258017    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258020    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258022    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 23:07:31.261192 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258025    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258417    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258423    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258426    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258429    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258432    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258435    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258437    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258440    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258442    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258445    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258447    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258450    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258452    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258455    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258458    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258461    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258464    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258466    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258469    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 23:07:31.261676 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258472    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258474    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258477    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258480    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258482    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258486    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258489    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258492    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258495    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258498    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258500    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258503    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258505    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258508    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258510    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258513    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258515    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258518    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258521    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 23:07:31.262171 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258523    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258526    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258528    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258530    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258533    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258535    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258538    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258540    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258543    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258549    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258551    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258554    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258556    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258559    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258561    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258564    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258566    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258569    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258571    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258573    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 23:07:31.262630 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258576    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258579    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258581    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258584    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258586    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258589    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258591    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258594    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258598    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258601    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258604    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258607    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258610    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258612    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258615    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258618    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258621    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258624    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258627    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 23:07:31.263128 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258629    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258632    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258639    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258641    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258644    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258646    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258649    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258651    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258653    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258656    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258660    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258663    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258666    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258669    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258672    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258674    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258676    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258679    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258681    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258684    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 23:07:31.263578 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258687    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.258690    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258771    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258780    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258790    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258797    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258803    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258806    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258811    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258815    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258818    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258822    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258825    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258828    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258835    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258839    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258841    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258844    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258848    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258851    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258855    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258858    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258861    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258864    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 23:07:31.264064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258867    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258870    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258873    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258876    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258879    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258882    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258886    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258888    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258891    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258894    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258898    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258901    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258904    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258907    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258910    2577 flags.go:64] FLAG: --enable-server="true"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258913    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258918    2577 flags.go:64] FLAG: --event-burst="100"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258921    2577 flags.go:64] FLAG: --event-qps="50"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258924    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258927    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258929    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258933    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258938    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258941    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258944    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 08 23:07:31.264622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258947    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258950    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258953    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258956    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258959    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258961    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258965    2577 flags.go:64] FLAG: --feature-gates=""
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258968    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258971    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258974    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258977    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258980    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258983    2577 flags.go:64] FLAG: --help="false"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258986    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258989    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258992    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258995    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.258998    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259001    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259004    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259007    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259009    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259012    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 23:07:31.265219 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259015    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259018    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259021    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259024    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259027    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259029    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259034    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259037    2577 flags.go:64] FLAG: --lock-file=""
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259040    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259043    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259046    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259051    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259054    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259056    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259059    2577 flags.go:64] FLAG: --logging-format="text"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259062    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259065    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259068    2577 flags.go:64] FLAG: --manifest-url=""
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259070    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259079    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259082    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259086    2577 flags.go:64] FLAG: --max-pods="110"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259089    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259091    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259094    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 23:07:31.265850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259097    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259100    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259103    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259106    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259114    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259117    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259120    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259123    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259126    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259131    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259133    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259136    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259139    2577 flags.go:64] FLAG: --port="10250"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259143    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259148    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-068a98ac2b56c0832"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259151    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259154    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259157    2577 flags.go:64] FLAG: --register-node="true"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259159    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259163    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259166    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259169    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259171    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259175    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259178    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 23:07:31.266867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259181    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259184    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259187    2577 flags.go:64] FLAG: --runonce="false"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259189    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259192    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259195    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259198    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259200    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259203    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259206    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259209    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259212    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259215    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259218    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259220    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259224    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259226    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259229    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259234    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259237    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259244    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259249    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259252    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259255    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259258    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 23:07:31.267490 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259260    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259263    2577 flags.go:64] FLAG: --v="2"
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259267    2577 flags.go:64] FLAG: --version="false"
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259271    2577 flags.go:64] FLAG: --vmodule=""
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259274    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.259278    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259391    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259395    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259398    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259401    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259404    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259407    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259409    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259412    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259414    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259417    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259420    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259422    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259425    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259428    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259430    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 23:07:31.268109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259433    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259435    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259438    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259441    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259445    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259449    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259454    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259457    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259460    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259462    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259464    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259467    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259470    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259472    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259475    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259477    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259480    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259482    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259485    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 23:07:31.268620 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259487    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259490    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259492    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259495    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259498    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259500    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259503    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259505    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259508    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259510    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259513    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259515    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259518    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259520    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259522    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259527    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259530    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259533    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259536    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259540    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 23:07:31.269093 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259543    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259546    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259549    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259552    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259554    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259557    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259560    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259562    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259565    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259567    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259569    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259572    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259574    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259577    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259580    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259582    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259585    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259587    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259590    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259592    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 23:07:31.269593 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259595    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259597    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259599    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259602    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259604    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259607    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259609    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259612    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259614    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259617    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259620    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259624    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259626    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259629    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259631    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259634    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259636    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259639    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259641    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259643    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 23:07:31.270078 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259646    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259649    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259653    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259656    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.259659    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.260285    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268013    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268028    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268041    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268047    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268103    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268108    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268112    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268115    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 23:07:31.270566 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268118    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268122    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268125    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268128    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268131    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268133    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268136    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268139    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268141    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268144    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268146    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268149    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268152    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268154    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268157    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268159    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268162    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268165    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268167    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268170    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 23:07:31.270939 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268172    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268175    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268178    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268181    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268184    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268188    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268190    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268193    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268196    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268198    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268201    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268203    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268206    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268209    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268211    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268214    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268216    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268219    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268221    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268224    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 23:07:31.271406 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268226    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268229    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268232    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268235    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268238    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268240    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268243    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268245    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268248    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268251    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268253    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268255    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268258    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268261    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268264    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268266    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268269    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268272    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268274    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268277    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 23:07:31.271887 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268279    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268282    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268285    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268289    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268292    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268295    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268298    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268301    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268303    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268306    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268308    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268311    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268313    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268317    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268321    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268325    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268327    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268330    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268333    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 23:07:31.272356 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268336    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268338    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268341    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268343    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268346    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268348    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268351    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268354    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268358    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268361    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268364    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268367    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268371    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268373    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268376    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 23:07:31.272834 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268379    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268384    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268492    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268497    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268500    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268503    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268506    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268509    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268512    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268514    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268517    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268520    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268522    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268525    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268528    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 23:07:31.273205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268531    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268533    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268536    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268539    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268541    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268544    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268547    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268549    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268552    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268554    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268557    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268559    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268562    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268564    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268566    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268569    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268571    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268574    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268577    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268579    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 23:07:31.273564 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268581    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268584    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268586    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268589    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268591    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268594    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268597    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268600    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268602    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268604    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268607    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268609    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268613    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268615    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268618    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268620    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268623    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268626    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268629    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 23:07:31.274046 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268632    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268635    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268637    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268640    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268642    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268644    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268647    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268649    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268652    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268654    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268657    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268659    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268661    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268664    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268667    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268669    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268671    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268674    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268676    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268679    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 23:07:31.274503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268681    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268684    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268686    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268688    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268691    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268694    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268697    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268713    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268716    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268718    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268721    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268725    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268728    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268731    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268734    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268737    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268740    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268742    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268745    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 23:07:31.275147 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268747    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268751    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268755    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268757    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268760    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268763    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268766    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:31.268769    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268774    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.268876    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.271090    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.272290    2577 server.go:1019] "Starting client certificate rotation"
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.272393    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 23:07:31.275601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.272428    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 23:07:31.275955 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.275413    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 23:07:31.278859 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.278842    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 23:07:31.290433 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.290417    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 08 23:07:31.292985 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.292965    2577 log.go:25] "Validated CRI v1 image API"
Jul 08 23:07:31.293760 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.293745    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 23:07:31.296255 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.296239    2577 fs.go:135] Filesystem UUIDs: map[463de52c-8f16-445a-82d2-579ce2b9217b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d785f287-69d0-4d95-bb11-accbf0fd00a3:/dev/nvme0n1p4]
Jul 08 23:07:31.296316 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.296255    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 23:07:31.299255 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.299230    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 23:07:31.301974 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.301868    2577 manager.go:216] Machine: {Timestamp:2026-07-08 23:07:31.300849494 +0000 UTC m=+0.292682791 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112338 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22d4bc5600fa8094c416aaa837ed50 SystemUUID:ec22d4bc-5600-fa80-94c4-16aaa837ed50 BootID:fd3d7dac-1ecf-4721-bfd2-4b2239f0ceac Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:7b:03:7d:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:7b:03:7d:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:67:8c:25:df:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 23:07:31.301974 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.301967    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 23:07:31.302097 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.302044    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 23:07:31.304318 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.304296    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 23:07:31.304442 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.304321    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 23:07:31.305092 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.305082    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 23:07:31.305129 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.305095    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 23:07:31.305129 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.305103    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 23:07:31.305730 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.305718    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 23:07:31.307333 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.307323    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 23:07:31.310485 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310473    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 23:07:31.310593 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310583    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 23:07:31.310629 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310613    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 23:07:31.310629 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310622    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 23:07:31.310682 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310633    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 23:07:31.310682 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310642    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 08 23:07:31.310682 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310652    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 23:07:31.310989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.310975    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 23:07:31.311042 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.311002    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 23:07:31.313104 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.313089    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 23:07:31.315211 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.315195    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 23:07:31.315290 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.315233    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 23:07:31.316338 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316316    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 23:07:31.316338 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316335    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316345    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316353    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316368    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316377    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316398    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316407    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316416    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316424    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316445    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 23:07:31.316459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.316454    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 23:07:31.317556 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.317545    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 23:07:31.317609 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.317558    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 23:07:31.320067 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.320044    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 23:07:31.320176 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.320160    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 23:07:31.320239 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.320158    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 23:07:31.320549 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.320533    2577 server.go:1268] "Started kubelet"
Jul 08 23:07:31.320839 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.320819    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 23:07:31.321081 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.321033    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 23:07:31.321155 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.321119    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 23:07:31.322316 ip-10-0-130-115 systemd[1]: Started Kubernetes Kubelet.
Jul 08 23:07:31.322898 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.322882    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 23:07:31.323417 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.323405    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 23:07:31.324306 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.324285    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7xqfn"
Jul 08 23:07:31.325738 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.325718    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 23:07:31.326741 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.325912    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-115.ec2.internal.18c073241960c69d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-115.ec2.internal,UID:ip-10-0-130-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-115.ec2.internal,},FirstTimestamp:2026-07-08 23:07:31.320506013 +0000 UTC m=+0.312339310,LastTimestamp:2026-07-08 23:07:31.320506013 +0000 UTC m=+0.312339310,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-115.ec2.internal,}"
Jul 08 23:07:31.329488 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.329470    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 23:07:31.329563 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.329473    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 23:07:31.329719 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.329691    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.329884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.329867    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7xqfn"
Jul 08 23:07:31.330247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330071    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 23:07:31.330247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330074    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 23:07:31.330247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330102    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 23:07:31.330247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330208    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 23:07:31.330247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330221    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 23:07:31.330495 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330404    2577 factory.go:55] Registering systemd factory
Jul 08 23:07:31.330495 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330420    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 08 23:07:31.330647 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330630    2577 factory.go:153] Registering CRI-O factory
Jul 08 23:07:31.330647 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330647    2577 factory.go:223] Registration of the crio container factory successfully
Jul 08 23:07:31.330784 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330693    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 23:07:31.330784 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330731    2577 factory.go:103] Registering Raw factory
Jul 08 23:07:31.330784 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.330743    2577 manager.go:1196] Started watching for new ooms in manager
Jul 08 23:07:31.331275 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.331260    2577 manager.go:349] Starting recovery of all containers
Jul 08 23:07:31.335395 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.335370    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 23:07:31.338016 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.337991    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-115.ec2.internal\" not found" node="ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.343994 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.343976    2577 manager.go:354] Recovery completed
Jul 08 23:07:31.348414 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.348402    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.350321 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350306    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.350386 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350336    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.350386 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350348    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.350893 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350881    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 23:07:31.350893 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350888    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.350965 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350894    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 23:07:31.350965 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.350914    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 23:07:31.351540 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.351521    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.351609 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.351552    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.351609 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.351564    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.352003 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.351989    2577 policy_none.go:49] "None policy: Start"
Jul 08 23:07:31.352003 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.352004    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 23:07:31.352189 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.352015    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 23:07:31.352994 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.352982    2577 policy_none.go:47] "Start"
Jul 08 23:07:31.376439 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376309    2577 manager.go:337] "Starting Device Plugin manager"
Jul 08 23:07:31.376536 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.376454    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 23:07:31.376536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376468    2577 server.go:87] "Starting device plugin registration server"
Jul 08 23:07:31.376667 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376656    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 23:07:31.376737 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376669    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 23:07:31.376797 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376757    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 23:07:31.376900 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376882    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 23:07:31.376900 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.376896    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 23:07:31.377753 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.377735    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 23:07:31.377823 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.377788    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.477188 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.477162    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.478248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.478134    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.478248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.478163    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.478248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.478174    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.478248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.478215    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.485837 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.485820    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.485880 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.485842    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-115.ec2.internal\": node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.499515 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.499456    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.512550 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.512523    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 23:07:31.513655 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.513638    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 23:07:31.513749 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.513673    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 23:07:31.513749 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.513692    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 23:07:31.513749 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.513741    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 23:07:31.515945 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.515925    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 23:07:31.602574 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.602505    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.614528 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.614496    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"]
Jul 08 23:07:31.614598 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.614588    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.616980 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.616960    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.617083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.616997    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.617083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.617012    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.618345 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.618329    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.618457 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.618441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.618521 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.618473    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.619412 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.619396    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.619412 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.619405    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.619578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.619429    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.619578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.619432    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.619578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.619444    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.619578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.619447    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.620587 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.620571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.620659 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.620598    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 23:07:31.622298 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.622280    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 23:07:31.622416 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.622307    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 23:07:31.622416 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.622319    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeHasSufficientPID"
Jul 08 23:07:31.631656 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.631638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b46bf19a4c8c74cc56a4f755de60986-config\") pod \"kube-apiserver-proxy-ip-10-0-130-115.ec2.internal\" (UID: \"4b46bf19a4c8c74cc56a4f755de60986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.631735 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.631666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.631735 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.631684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.644557 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.644537    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-115.ec2.internal\" not found" node="ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.649487 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.649473    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-115.ec2.internal\" not found" node="ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.702633 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.702598    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.732821 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.732791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.732901 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.732827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.732901 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.732853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b46bf19a4c8c74cc56a4f755de60986-config\") pod \"kube-apiserver-proxy-ip-10-0-130-115.ec2.internal\" (UID: \"4b46bf19a4c8c74cc56a4f755de60986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.732901 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.732891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.733001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.732897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4b46bf19a4c8c74cc56a4f755de60986-config\") pod \"kube-apiserver-proxy-ip-10-0-130-115.ec2.internal\" (UID: \"4b46bf19a4c8c74cc56a4f755de60986\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.733001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.732901    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc08c5a47b8f2a9d0e8afb0797a72354-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal\" (UID: \"fc08c5a47b8f2a9d0e8afb0797a72354\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.804411 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.804330    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.909985 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:31.909935    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:31.948216 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.948184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:31.954164 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:31.953960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:32.011126 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:32.011092    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:32.116630 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:32.116542    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:32.217420 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:32.217388    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:32.272083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.272053    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 23:07:32.272535 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.272219    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 23:07:32.272535 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.272244    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 23:07:32.318410 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:32.318381    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:32.331868 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.331816    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 23:02:32 +0000 UTC" deadline="2028-02-21 02:50:06.906461972 +0000 UTC"
Jul 08 23:07:32.331868 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.331862    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14211h42m34.574602854s"
Jul 08 23:07:32.362312 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.362286    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 23:07:32.418554 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:32.418489    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-115.ec2.internal\" not found"
Jul 08 23:07:32.432570 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.432549    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 23:07:32.530214 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.530196    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:32.541941 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.541907    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 23:07:32.542914 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.542895    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal"
Jul 08 23:07:32.551604 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.551586    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 23:07:32.667054 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:32.667002    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b46bf19a4c8c74cc56a4f755de60986.slice/crio-9fdb20c9dc7aabb669a36cdc5e0875c80cb2d6a073febb9a522bc4373797de3b WatchSource:0}: Error finding container 9fdb20c9dc7aabb669a36cdc5e0875c80cb2d6a073febb9a522bc4373797de3b: Status 404 returned error can't find the container with id 9fdb20c9dc7aabb669a36cdc5e0875c80cb2d6a073febb9a522bc4373797de3b
Jul 08 23:07:32.668021 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:32.667993    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc08c5a47b8f2a9d0e8afb0797a72354.slice/crio-dd7c8e3127a3b453f53eeeaeae3ebd650c56aea56179ecd56d511616a416aaaa WatchSource:0}: Error finding container dd7c8e3127a3b453f53eeeaeae3ebd650c56aea56179ecd56d511616a416aaaa: Status 404 returned error can't find the container with id dd7c8e3127a3b453f53eeeaeae3ebd650c56aea56179ecd56d511616a416aaaa
Jul 08 23:07:32.671901 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:32.671879    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 23:07:33.313045 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.312814    2577 apiserver.go:52] "Watching apiserver"
Jul 08 23:07:33.320827 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.320170    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 23:07:33.320827 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.320729    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-zftwc","kube-system/global-pull-secret-syncer-x4psn","kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal","openshift-cluster-node-tuning-operator/tuned-brmfm","openshift-dns/node-resolver-hj86f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-vd9d9","openshift-network-diagnostics/network-check-target-8dfb9","kube-system/konnectivity-agent-zwhlx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4","openshift-image-registry/node-ca-45jkz","openshift-multus/multus-gzvzj","openshift-multus/network-metrics-daemon-ftnp8","openshift-network-operator/iptables-alerter-6v9qm"]
Jul 08 23:07:33.323856 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.323832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.326622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.326324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.326622 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.326444    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:33.327428 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.327176    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 23:07:33.327428 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.327249    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 23:07:33.327428 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.327274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.327428 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.327295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 23:07:33.327752 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.327531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.327811 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.327738    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmtgk\""
Jul 08 23:07:33.329544 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.329103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.329851 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.329811    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 23:07:33.335479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.334769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.335479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.335311    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.335479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.335322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.335479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.335345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 23:07:33.335730 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.335535    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 23:07:33.336693 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.336523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qdjpd\""
Jul 08 23:07:33.337731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.337271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.337731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.337538    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9sls2\""
Jul 08 23:07:33.338405 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.338213    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.338405 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.338372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.340927 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.340573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:33.340927 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.340650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:33.340927 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.340865    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.340927 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.340913    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7ltzd\""
Jul 08 23:07:33.341268 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.341227    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.341479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.341457    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 23:07:33.343092 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.343412 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-modprobe-d\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.343486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-lib-modules\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343600 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-cnibin\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.343600 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2522b38-49ba-4621-8a65-190d5a4f7ad2-host-slash\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.343600 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343534    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvlr\" (UniqueName: \"kubernetes.io/projected/f2522b38-49ba-4621-8a65-190d5a4f7ad2-kube-api-access-qqvlr\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.343600 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysconfig\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343834 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-kubernetes\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343834 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysctl-conf\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343834 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-run\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343834 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysctl-d\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343834 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-sys\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.343834 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-host\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70527739-484e-49b9-bfad-4b178e0af08a-etc-tuned\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2522b38-49ba-4621-8a65-190d5a4f7ad2-iptables-alerter-script\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70527739-484e-49b9-bfad-4b178e0af08a-tmp\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.343993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-cni-binary-copy\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-var-lib-kubelet\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-system-cni-dir\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-os-release\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52s4\" (UniqueName: \"kubernetes.io/projected/753217f0-e74b-40cf-a22f-6ddb1e4af404-kube-api-access-f52s4\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.344146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjsp\" (UniqueName: \"kubernetes.io/projected/70527739-484e-49b9-bfad-4b178e0af08a-kube-api-access-lrjsp\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.344586 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.344586 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-systemd\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.344586 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.344586 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.344338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-kube-api-access-tfnmw\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.345499 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.345476    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 23:07:33.345770 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.345685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 23:07:33.345770 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.345701    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cm4j2\""
Jul 08 23:07:33.346814 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.346726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.350132 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.350112    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.350322 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.350307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.350501 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.350485    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v7dqn\""
Jul 08 23:07:33.350685 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.350670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 23:07:33.350889 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.350874    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 23:07:33.351069 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.351050    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 23:07:33.351273 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.351256    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 23:07:33.351518 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.351500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.351599 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.351576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:33.355006 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.354832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.357260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.357179    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.357563 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.357497    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s5526\""
Jul 08 23:07:33.357563 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.357520    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 23:07:33.357694 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.357621    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.357815 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.357794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.360332 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.360288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lf6ss\""
Jul 08 23:07:33.360332 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.360317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 23:07:33.360746 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.360584    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 23:07:33.360861 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.360773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 23:07:33.361255 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.361235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.364004 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.363972    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvx6k"
Jul 08 23:07:33.364086 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.364072    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c8x9n\""
Jul 08 23:07:33.364282 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.364266    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 23:07:33.372230 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.372204    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mvx6k"
Jul 08 23:07:33.431224 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.431203    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 23:07:33.443081 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.443058    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 23:07:33.444946 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.444922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-modprobe-d\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.444961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.445043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.444993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-systemd\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.445043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj4xx\" (UniqueName: \"kubernetes.io/projected/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-kube-api-access-bj4xx\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbf63736-facd-4dd4-ac3e-418bc5356176-kubelet-config\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kms5z\" (UniqueName: \"kubernetes.io/projected/88b0968b-f35e-48d1-9e4a-62d157eda27b-kube-api-access-kms5z\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-lib-modules\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-run\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-kubelet\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-etc-kubernetes\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysctl-d\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-sys\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-node-log\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-cni-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445292    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-lib-modules\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-k8s-cni-cncf-io\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b38f8-7c7a-4f3e-bbe7-78a70800814a-konnectivity-ca\") pod \"konnectivity-agent-zwhlx\" (UID: \"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a\") " pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2522b38-49ba-4621-8a65-190d5a4f7ad2-iptables-alerter-script\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70527739-484e-49b9-bfad-4b178e0af08a-tmp\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-cni-binary-copy\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-var-lib-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445472    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysctl-d\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-netns\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-sys-fs\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-var-lib-kubelet\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-os-release\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52s4\" (UniqueName: \"kubernetes.io/projected/753217f0-e74b-40cf-a22f-6ddb1e4af404-kube-api-access-f52s4\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.445714 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-socket-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445651    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-cnibin\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-hosts-file\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445803    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-run\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-systemd\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-slash\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445870    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-var-lib-kubelet\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-etc-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-system-cni-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445915    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445933    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-systemd\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88b0968b-f35e-48d1-9e4a-62d157eda27b-host\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.445947    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-os-release\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446119    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f2522b38-49ba-4621-8a65-190d5a4f7ad2-iptables-alerter-script\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446179    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-modprobe-d\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.446581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-systemd-units\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88b0968b-f35e-48d1-9e4a-62d157eda27b-serviceca\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446250    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-sys\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-log-socket\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-run-ovn-kubernetes\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-socket-dir-parent\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-cni-bin\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbf63736-facd-4dd4-ac3e-418bc5356176-dbus\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-cnibin\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2522b38-49ba-4621-8a65-190d5a4f7ad2-host-slash\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvlr\" (UniqueName: \"kubernetes.io/projected/f2522b38-49ba-4621-8a65-190d5a4f7ad2-kube-api-access-qqvlr\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446508    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-cni-binary-copy\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysconfig\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-cnibin\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-kubernetes\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysctl-conf\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446578    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/753217f0-e74b-40cf-a22f-6ddb1e4af404-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.447548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-host\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysconfig\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70527739-484e-49b9-bfad-4b178e0af08a-etc-tuned\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-cni-bin\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-kubernetes\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-cni-netd\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5feb9d3-7783-4adf-add9-e666e310c820-ovn-node-metrics-cert\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-etc-sysctl-conf\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446716    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70527739-484e-49b9-bfad-4b178e0af08a-host\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446575    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2522b38-49ba-4621-8a65-190d5a4f7ad2-host-slash\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2tj\" (UniqueName: \"kubernetes.io/projected/c5feb9d3-7783-4adf-add9-e666e310c820-kube-api-access-sh2tj\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/476b5c69-63b7-4a24-8843-a0a97efe69c8-cni-binary-copy\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-hostroot\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-device-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-etc-selinux\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-registration-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.446978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d9kq\" (UniqueName: \"kubernetes.io/projected/8eca8787-38ab-436c-9e9f-eeda050cf706-kube-api-access-6d9kq\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.448464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-os-release\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-kubelet\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-conf-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb5b38f8-7c7a-4f3e-bbe7-78a70800814a-agent-certs\") pod \"konnectivity-agent-zwhlx\" (UID: \"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a\") " pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-system-cni-dir\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-env-overrides\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxp4\" (UniqueName: \"kubernetes.io/projected/476b5c69-63b7-4a24-8843-a0a97efe69c8-kube-api-access-dcxp4\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-ovnkube-script-lib\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjsp\" (UniqueName: \"kubernetes.io/projected/70527739-484e-49b9-bfad-4b178e0af08a-kube-api-access-lrjsp\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-run-netns\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-ovnkube-config\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-daemon-config\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-multus-certs\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-kube-api-access-tfnmw\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.449350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-ovn\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447434    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447490    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/753217f0-e74b-40cf-a22f-6ddb1e4af404-system-cni-dir\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-cni-multus\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.447637    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-tmp-dir\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.447687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.450248 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.447712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:33.947690577 +0000 UTC m=+2.939523865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:33.450661 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.450479    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70527739-484e-49b9-bfad-4b178e0af08a-tmp\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.450717 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.450654    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70527739-484e-49b9-bfad-4b178e0af08a-etc-tuned\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.457309 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.457253    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvlr\" (UniqueName: \"kubernetes.io/projected/f2522b38-49ba-4621-8a65-190d5a4f7ad2-kube-api-access-qqvlr\") pod \"iptables-alerter-6v9qm\" (UID: \"f2522b38-49ba-4621-8a65-190d5a4f7ad2\") " pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.457598 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.457574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52s4\" (UniqueName: \"kubernetes.io/projected/753217f0-e74b-40cf-a22f-6ddb1e4af404-kube-api-access-f52s4\") pod \"multus-additional-cni-plugins-vd9d9\" (UID: \"753217f0-e74b-40cf-a22f-6ddb1e4af404\") " pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.459691 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.459670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjsp\" (UniqueName: \"kubernetes.io/projected/70527739-484e-49b9-bfad-4b178e0af08a-kube-api-access-lrjsp\") pod \"tuned-brmfm\" (UID: \"70527739-484e-49b9-bfad-4b178e0af08a\") " pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.459808 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.459788    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnmw\" (UniqueName: \"kubernetes.io/projected/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-kube-api-access-tfnmw\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.519717 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.519618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" event={"ID":"fc08c5a47b8f2a9d0e8afb0797a72354","Type":"ContainerStarted","Data":"dd7c8e3127a3b453f53eeeaeae3ebd650c56aea56179ecd56d511616a416aaaa"}
Jul 08 23:07:33.520978 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.520952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" event={"ID":"4b46bf19a4c8c74cc56a4f755de60986","Type":"ContainerStarted","Data":"9fdb20c9dc7aabb669a36cdc5e0875c80cb2d6a073febb9a522bc4373797de3b"}
Jul 08 23:07:33.548817 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-node-log\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.548947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-cni-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.548947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-k8s-cni-cncf-io\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.548947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548901    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-k8s-cni-cncf-io\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.548947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-node-log\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b38f8-7c7a-4f3e-bbe7-78a70800814a-konnectivity-ca\") pod \"konnectivity-agent-zwhlx\" (UID: \"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a\") " pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.548988    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-cni-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-var-lib-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-netns\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-sys-fs\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-socket-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549108    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549129    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-netns\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549162 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549145    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-var-lib-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-sys-fs\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-cnibin\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-hosts-file\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549234    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-socket-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-cnibin\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-slash\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549372    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-hosts-file\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-etc-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549391    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-etc-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-system-cni-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549429    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-slash\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-system-cni-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88b0968b-f35e-48d1-9e4a-62d157eda27b-host\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-systemd-units\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88b0968b-f35e-48d1-9e4a-62d157eda27b-serviceca\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.549668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-systemd-units\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549508    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb5b38f8-7c7a-4f3e-bbe7-78a70800814a-konnectivity-ca\") pod \"konnectivity-agent-zwhlx\" (UID: \"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a\") " pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549512    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88b0968b-f35e-48d1-9e4a-62d157eda27b-host\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-log-socket\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-run-ovn-kubernetes\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549613    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-log-socket\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-socket-dir-parent\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549653    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-run-ovn-kubernetes\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-cni-bin\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbf63736-facd-4dd4-ac3e-418bc5356176-dbus\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-socket-dir-parent\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-cni-bin\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-cni-netd\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549816    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-cni-bin\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-cni-netd\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549833    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbf63736-facd-4dd4-ac3e-418bc5356176-dbus\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5feb9d3-7783-4adf-add9-e666e310c820-ovn-node-metrics-cert\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549887    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-cni-bin\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.550506 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.549988    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88b0968b-f35e-48d1-9e4a-62d157eda27b-serviceca\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2tj\" (UniqueName: \"kubernetes.io/projected/c5feb9d3-7783-4adf-add9-e666e310c820-kube-api-access-sh2tj\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/476b5c69-63b7-4a24-8843-a0a97efe69c8-cni-binary-copy\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-hostroot\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550110    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-hostroot\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-device-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-etc-selinux\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-registration-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d9kq\" (UniqueName: \"kubernetes.io/projected/8eca8787-38ab-436c-9e9f-eeda050cf706-kube-api-access-6d9kq\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550404    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-device-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-os-release\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-kubelet\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550466    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-registration-dir\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-conf-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb5b38f8-7c7a-4f3e-bbe7-78a70800814a-agent-certs\") pod \"konnectivity-agent-zwhlx\" (UID: \"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a\") " pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550530    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550552    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8eca8787-38ab-436c-9e9f-eeda050cf706-etc-selinux\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.551459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-env-overrides\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxp4\" (UniqueName: \"kubernetes.io/projected/476b5c69-63b7-4a24-8843-a0a97efe69c8-kube-api-access-dcxp4\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550599    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-kubelet\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-ovnkube-script-lib\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550662    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-conf-dir\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-run-netns\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-ovnkube-config\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-daemon-config\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-multus-certs\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-ovn\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-cni-multus\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-tmp-dir\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550869    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-os-release\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550913    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-openvswitch\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-systemd\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj4xx\" (UniqueName: \"kubernetes.io/projected/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-kube-api-access-bj4xx\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.552492 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.551004    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbf63736-facd-4dd4-ac3e-418bc5356176-kubelet-config\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-systemd\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kms5z\" (UniqueName: \"kubernetes.io/projected/88b0968b-f35e-48d1-9e4a-62d157eda27b-kube-api-access-kms5z\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551095    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-run-ovn\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.551110    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:34.051092508 +0000 UTC m=+3.042925799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-kubelet\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-etc-kubernetes\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551242    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbf63736-facd-4dd4-ac3e-418bc5356176-kubelet-config\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551257    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-etc-kubernetes\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551525    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-kubelet\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551565    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-env-overrides\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551588    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-run-multus-certs\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551694    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-tmp-dir\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551701    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/476b5c69-63b7-4a24-8843-a0a97efe69c8-multus-daemon-config\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.550621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/476b5c69-63b7-4a24-8843-a0a97efe69c8-cni-binary-copy\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551744    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-ovnkube-config\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.553502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551753    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/476b5c69-63b7-4a24-8843-a0a97efe69c8-host-var-lib-cni-multus\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.554435 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.551814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5feb9d3-7783-4adf-add9-e666e310c820-host-run-netns\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.554435 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.552109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5feb9d3-7783-4adf-add9-e666e310c820-ovnkube-script-lib\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.554435 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.553534    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5feb9d3-7783-4adf-add9-e666e310c820-ovn-node-metrics-cert\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.554599 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.554513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb5b38f8-7c7a-4f3e-bbe7-78a70800814a-agent-certs\") pod \"konnectivity-agent-zwhlx\" (UID: \"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a\") " pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.558407 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.558095    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:07:33.558407 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.558121    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:07:33.558407 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.558134    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:33.558407 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.558249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:34.058204871 +0000 UTC m=+3.050038162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:33.559295 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.559270    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2tj\" (UniqueName: \"kubernetes.io/projected/c5feb9d3-7783-4adf-add9-e666e310c820-kube-api-access-sh2tj\") pod \"ovnkube-node-zftwc\" (UID: \"c5feb9d3-7783-4adf-add9-e666e310c820\") " pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.561339 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.561295    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d9kq\" (UniqueName: \"kubernetes.io/projected/8eca8787-38ab-436c-9e9f-eeda050cf706-kube-api-access-6d9kq\") pod \"aws-ebs-csi-driver-node-2s6d4\" (UID: \"8eca8787-38ab-436c-9e9f-eeda050cf706\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.562121 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.561929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxp4\" (UniqueName: \"kubernetes.io/projected/476b5c69-63b7-4a24-8843-a0a97efe69c8-kube-api-access-dcxp4\") pod \"multus-gzvzj\" (UID: \"476b5c69-63b7-4a24-8843-a0a97efe69c8\") " pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.562121 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.562079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kms5z\" (UniqueName: \"kubernetes.io/projected/88b0968b-f35e-48d1-9e4a-62d157eda27b-kube-api-access-kms5z\") pod \"node-ca-45jkz\" (UID: \"88b0968b-f35e-48d1-9e4a-62d157eda27b\") " pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.562121 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.562113    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj4xx\" (UniqueName: \"kubernetes.io/projected/2c5b15c7-69bb-496a-9080-bb4ac43ae4fb-kube-api-access-bj4xx\") pod \"node-resolver-hj86f\" (UID: \"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb\") " pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.638796 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.638717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vd9d9"
Jul 08 23:07:33.653267 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.653247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6v9qm"
Jul 08 23:07:33.663247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.663228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-brmfm"
Jul 08 23:07:33.669036 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.669017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hj86f"
Jul 08 23:07:33.678981 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.678965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gzvzj"
Jul 08 23:07:33.686781 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.686760    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:33.696998 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.696975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:07:33.704131 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.704112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4"
Jul 08 23:07:33.710111 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.710088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-45jkz"
Jul 08 23:07:33.954721 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:33.954642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:33.954917 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.954798    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:33.954917 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:33.954850    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:34.954836257 +0000 UTC m=+3.946669542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:34.055000 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.054961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:34.055177 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.055124    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:34.055240 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.055194    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:35.055176431 +0000 UTC m=+4.047009722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:34.155938 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.155905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:34.156105 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.156065    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:07:34.156105 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.156083    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:07:34.156105 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.156092    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:34.156215 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.156146    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:35.156129651 +0000 UTC m=+4.147962947 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:34.376244 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.376146    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 23:02:34 +0000 UTC" deadline="2028-04-08 14:24:55.617400808 +0000 UTC"
Jul 08 23:07:34.376244 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.376197    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15351h17m21.241209079s"
Jul 08 23:07:34.514120 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.514079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:34.514293 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.514213    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:34.514293 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.514278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:34.514455 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.514422    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:34.741827 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.741793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70527739_484e_49b9_bfad_4b178e0af08a.slice/crio-bf94d61a585208899c2370959c241008e7ae83fca7e63aca8f7d3c66d694672f WatchSource:0}: Error finding container bf94d61a585208899c2370959c241008e7ae83fca7e63aca8f7d3c66d694672f: Status 404 returned error can't find the container with id bf94d61a585208899c2370959c241008e7ae83fca7e63aca8f7d3c66d694672f
Jul 08 23:07:34.744977 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.744855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b0968b_f35e_48d1_9e4a_62d157eda27b.slice/crio-104e6372553fa67ca853bca236aa00bf9b30e5f7bb4dd1be10555ba5264bf2d6 WatchSource:0}: Error finding container 104e6372553fa67ca853bca236aa00bf9b30e5f7bb4dd1be10555ba5264bf2d6: Status 404 returned error can't find the container with id 104e6372553fa67ca853bca236aa00bf9b30e5f7bb4dd1be10555ba5264bf2d6
Jul 08 23:07:34.748272 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.748248    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2522b38_49ba_4621_8a65_190d5a4f7ad2.slice/crio-1fb61073e223f52b365ad1f1f2d502c5a59e2842c3e3c49823e024a18fb8dcbd WatchSource:0}: Error finding container 1fb61073e223f52b365ad1f1f2d502c5a59e2842c3e3c49823e024a18fb8dcbd: Status 404 returned error can't find the container with id 1fb61073e223f52b365ad1f1f2d502c5a59e2842c3e3c49823e024a18fb8dcbd
Jul 08 23:07:34.749884 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.749855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5b38f8_7c7a_4f3e_bbe7_78a70800814a.slice/crio-e63a091478320a65f785708430b896bbe543186fbecfe16c1fb3c4cf25c5749a WatchSource:0}: Error finding container e63a091478320a65f785708430b896bbe543186fbecfe16c1fb3c4cf25c5749a: Status 404 returned error can't find the container with id e63a091478320a65f785708430b896bbe543186fbecfe16c1fb3c4cf25c5749a
Jul 08 23:07:34.751608 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.751493    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5feb9d3_7783_4adf_add9_e666e310c820.slice/crio-d4aa675d146c7ca61512887afed53fd6521efc04238d57786e5e90d0cdf4faa0 WatchSource:0}: Error finding container d4aa675d146c7ca61512887afed53fd6521efc04238d57786e5e90d0cdf4faa0: Status 404 returned error can't find the container with id d4aa675d146c7ca61512887afed53fd6521efc04238d57786e5e90d0cdf4faa0
Jul 08 23:07:34.752424 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.752311    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eca8787_38ab_436c_9e9f_eeda050cf706.slice/crio-8534b1808383c17a895d870e83e1a07717ea5f74f4794f699d1cb42479ba9fc1 WatchSource:0}: Error finding container 8534b1808383c17a895d870e83e1a07717ea5f74f4794f699d1cb42479ba9fc1: Status 404 returned error can't find the container with id 8534b1808383c17a895d870e83e1a07717ea5f74f4794f699d1cb42479ba9fc1
Jul 08 23:07:34.753936 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.753261    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5b15c7_69bb_496a_9080_bb4ac43ae4fb.slice/crio-86e944147cd1b0fdc99fce715b2a3e28b244b41d2c78f90147dccc0cddafa496 WatchSource:0}: Error finding container 86e944147cd1b0fdc99fce715b2a3e28b244b41d2c78f90147dccc0cddafa496: Status 404 returned error can't find the container with id 86e944147cd1b0fdc99fce715b2a3e28b244b41d2c78f90147dccc0cddafa496
Jul 08 23:07:34.755500 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:07:34.755356    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753217f0_e74b_40cf_a22f_6ddb1e4af404.slice/crio-15d7768eacfb6ea5287ce35e26aaa9696a38467e41333c9250bbfbc6620f6122 WatchSource:0}: Error finding container 15d7768eacfb6ea5287ce35e26aaa9696a38467e41333c9250bbfbc6620f6122: Status 404 returned error can't find the container with id 15d7768eacfb6ea5287ce35e26aaa9696a38467e41333c9250bbfbc6620f6122
Jul 08 23:07:34.961775 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:34.961612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:34.961922 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.961792    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:34.961922 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:34.961861    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:36.961839708 +0000 UTC m=+5.953673001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:35.062575 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.062553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:35.062680 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.062664    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:35.062747 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.062731    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:37.062703082 +0000 UTC m=+6.054536367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:35.163446 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.163417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:35.163680 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.163553    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:07:35.163680 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.163578    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:07:35.163680 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.163594    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:35.163680 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.163654    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:37.163638147 +0000 UTC m=+6.155471438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:35.377177 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.377066    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 23:02:34 +0000 UTC" deadline="2027-12-21 07:01:22.439667067 +0000 UTC"
Jul 08 23:07:35.377177 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.377103    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12727h53m47.062568969s"
Jul 08 23:07:35.515179 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.515145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:35.515344 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:35.515287    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:35.537899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.537001    2577 generic.go:356] "Generic (PLEG): container finished" podID="fc08c5a47b8f2a9d0e8afb0797a72354" containerID="ef63506d6a7c11b6725756bf275b8516447952debfa0d5b7d38250e32b567bd9" exitCode=0
Jul 08 23:07:35.537899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.537082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" event={"ID":"fc08c5a47b8f2a9d0e8afb0797a72354","Type":"ContainerDied","Data":"ef63506d6a7c11b6725756bf275b8516447952debfa0d5b7d38250e32b567bd9"}
Jul 08 23:07:35.552770 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.552737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerStarted","Data":"15d7768eacfb6ea5287ce35e26aaa9696a38467e41333c9250bbfbc6620f6122"}
Jul 08 23:07:35.567088 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.567044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hj86f" event={"ID":"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb","Type":"ContainerStarted","Data":"86e944147cd1b0fdc99fce715b2a3e28b244b41d2c78f90147dccc0cddafa496"}
Jul 08 23:07:35.580614 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.580582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"d4aa675d146c7ca61512887afed53fd6521efc04238d57786e5e90d0cdf4faa0"}
Jul 08 23:07:35.597259 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.597186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6v9qm" event={"ID":"f2522b38-49ba-4621-8a65-190d5a4f7ad2","Type":"ContainerStarted","Data":"1fb61073e223f52b365ad1f1f2d502c5a59e2842c3e3c49823e024a18fb8dcbd"}
Jul 08 23:07:35.609404 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.609370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brmfm" event={"ID":"70527739-484e-49b9-bfad-4b178e0af08a","Type":"ContainerStarted","Data":"bf94d61a585208899c2370959c241008e7ae83fca7e63aca8f7d3c66d694672f"}
Jul 08 23:07:35.623615 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.623588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" event={"ID":"4b46bf19a4c8c74cc56a4f755de60986","Type":"ContainerStarted","Data":"06bacd6079b0726d54838335525220ab3201835af0aaa0c0bc902dd7be2d1ba9"}
Jul 08 23:07:35.637517 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.637455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gzvzj" event={"ID":"476b5c69-63b7-4a24-8843-a0a97efe69c8","Type":"ContainerStarted","Data":"1318e71d29bd6a5730cfdc61ece653242eeba29c87aa7ee2da4e67bb2257b6bd"}
Jul 08 23:07:35.667012 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.666986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4" event={"ID":"8eca8787-38ab-436c-9e9f-eeda050cf706","Type":"ContainerStarted","Data":"8534b1808383c17a895d870e83e1a07717ea5f74f4794f699d1cb42479ba9fc1"}
Jul 08 23:07:35.671969 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.671940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zwhlx" event={"ID":"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a","Type":"ContainerStarted","Data":"e63a091478320a65f785708430b896bbe543186fbecfe16c1fb3c4cf25c5749a"}
Jul 08 23:07:35.673633 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:35.673614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-45jkz" event={"ID":"88b0968b-f35e-48d1-9e4a-62d157eda27b","Type":"ContainerStarted","Data":"104e6372553fa67ca853bca236aa00bf9b30e5f7bb4dd1be10555ba5264bf2d6"}
Jul 08 23:07:36.514874 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:36.514840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:36.515357 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:36.514975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:36.515357 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:36.515351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:36.515479 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:36.515462    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:36.713415 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:36.713380    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" event={"ID":"fc08c5a47b8f2a9d0e8afb0797a72354","Type":"ContainerStarted","Data":"4c431c8a4b0572b571510b7f93bad413b0a66a0cf339c110e4b9ee1f7cf4e40f"}
Jul 08 23:07:36.730087 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:36.729978    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-115.ec2.internal" podStartSLOduration=3.72995457 podStartE2EDuration="3.72995457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:35.639170737 +0000 UTC m=+4.631004047" watchObservedRunningTime="2026-07-08 23:07:36.72995457 +0000 UTC m=+5.721787882"
Jul 08 23:07:36.978642 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:36.977939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:36.978642 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:36.978144    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:36.978642 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:36.978211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:40.978190195 +0000 UTC m=+9.970023481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:37.079511 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:37.078974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:37.079511 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.079106    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:37.079511 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.079166    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:41.079147562 +0000 UTC m=+10.070980850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:37.180852 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:37.180223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:37.180852 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.180396    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:07:37.180852 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.180418    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:07:37.180852 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.180430    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:37.180852 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.180489    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:41.180470081 +0000 UTC m=+10.172303369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:37.518950 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:37.518917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:37.519455 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:37.519053    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:38.515067 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:38.514334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:38.515067 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:38.514487    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:38.515067 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:38.514910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:38.515067 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:38.515017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:39.514413 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:39.514376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:39.514909 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:39.514517    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:40.514938 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:40.514904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:40.515400 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:40.515029    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:40.515400 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:40.515103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:40.515400 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:40.515253    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:41.012993 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:41.012898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:41.013154 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.013072    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:41.013226 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.013152    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:49.013128669 +0000 UTC m=+18.004961972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:41.113588 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:41.113559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:41.113768 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.113690    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:41.113843 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.113771    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:49.113735912 +0000 UTC m=+18.105569197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:41.215402 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:41.214736    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:41.215402 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.214923    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:07:41.215402 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.214945    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:07:41.215402 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.214959    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:41.215402 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.215023    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:07:49.215002195 +0000 UTC m=+18.206835484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:41.515409 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:41.515380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:41.515840 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:41.515483    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:42.514503 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:42.514458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:42.514700 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:42.514595    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:42.514700 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:42.514655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:42.514861 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:42.514806    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:43.514225 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:43.514179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:43.514725 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:43.514337    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:44.514538 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:44.514065    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:44.514538 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:44.514197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:44.514538 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:44.514366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:44.514538 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:44.514478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:45.514188 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:45.514160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:45.514371 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:45.514282    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:46.514618 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:46.514544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:46.515053 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:46.514664    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:46.515053 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:46.514727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:46.515053 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:46.514831    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:47.514385 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:47.514352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:47.514596 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:47.514481    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:48.514669 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:48.514638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:48.515199 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:48.514683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:48.515199 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:48.514795    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:48.515199 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:48.514950    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:49.076771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:49.076726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:49.076977 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.076903    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:49.077033 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.076977    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:05.076956017 +0000 UTC m=+34.068789306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:07:49.177738 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:49.177691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:49.177914 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.177817    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:49.177914 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.177886    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:05.177867889 +0000 UTC m=+34.169701180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:07:49.278426 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:49.278395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:49.278606 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.278575    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:07:49.278606 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.278595    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:07:49.278606 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.278603    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:49.278753 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.278649    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:05.278634866 +0000 UTC m=+34.270468150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:07:49.514367 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:49.514290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:49.514530 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:49.514412    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:50.514497 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:50.514468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:50.514951 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:50.514476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:50.514951 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:50.514584    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:50.514951 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:50.514696    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:51.515924 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:51.515894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:51.516357 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:51.516005    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:52.514619 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:52.514584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:52.514787 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:52.514587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:52.514787 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:52.514692    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:52.514903 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:52.514813    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:53.514719 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:53.514679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:53.515254 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:53.514814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:54.514683 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.514530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:54.514882 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.514602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:54.514882 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:54.514773    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:54.514882 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:54.514842    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:54.750071 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.750044    2577 generic.go:356] "Generic (PLEG): container finished" podID="753217f0-e74b-40cf-a22f-6ddb1e4af404" containerID="f41509ceb0657126e227eb970ac072432025e5dc755cc64394b997081b643c90" exitCode=0
Jul 08 23:07:54.750234 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.750112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerDied","Data":"f41509ceb0657126e227eb970ac072432025e5dc755cc64394b997081b643c90"}
Jul 08 23:07:54.751675 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.751642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hj86f" event={"ID":"2c5b15c7-69bb-496a-9080-bb4ac43ae4fb","Type":"ContainerStarted","Data":"f7631d5b1193ead0fad7fa4f549606fb214b3ad01ce04e1aa34d59564b3bdcb6"}
Jul 08 23:07:54.754744 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.754718    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"657091e5f44f1476fdc736a73bc3591bd6e57d09debdb4f0bf1fd104eb23e545"}
Jul 08 23:07:54.754862 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.754797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"cde1710aca259b3715c4f687735ae2f828362eb5622dae08e2709dcec228bb56"}
Jul 08 23:07:54.754862 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.754850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"938b9a146dd898fdc3fcf4fef4827c90ed3d3bb842121a2741d0e8083cb62205"}
Jul 08 23:07:54.755000 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.754882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"14d6ca0df0e421c4598b95f459fdb2244796ff1ccc1d7c76ea1641199bf8db8a"}
Jul 08 23:07:54.755000 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.754918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"26320b4fe655b4378d07abac2095820c683e78a22cb4042c7007c7239a155d61"}
Jul 08 23:07:54.755000 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.754946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"1f146254c87a2f02fd6611e1e99dac272ad3659819bea54c1d101822b57ffbcb"}
Jul 08 23:07:54.756128 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.756099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-brmfm" event={"ID":"70527739-484e-49b9-bfad-4b178e0af08a","Type":"ContainerStarted","Data":"e06a3f6f1697d1496ad08057563fb8f7a50a941c16a5de74eff305b86ea620ca"}
Jul 08 23:07:54.757499 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.757477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gzvzj" event={"ID":"476b5c69-63b7-4a24-8843-a0a97efe69c8","Type":"ContainerStarted","Data":"90f79225ea64399379cf8efab739e559f2b97f50c5f74ac89b4fd85cbb09668c"}
Jul 08 23:07:54.758827 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.758803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4" event={"ID":"8eca8787-38ab-436c-9e9f-eeda050cf706","Type":"ContainerStarted","Data":"99ee4ac744137061aacabebda4d854a7771a500268ef629049939fb62071b8bb"}
Jul 08 23:07:54.759889 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.759869    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zwhlx" event={"ID":"eb5b38f8-7c7a-4f3e-bbe7-78a70800814a","Type":"ContainerStarted","Data":"f18428e9469fb8e85f29608e3ad93f62f28be0d404bd06b8f606101b2e9a3a35"}
Jul 08 23:07:54.761047 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.761025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-45jkz" event={"ID":"88b0968b-f35e-48d1-9e4a-62d157eda27b","Type":"ContainerStarted","Data":"eed8758df7280a76a3289681351d4d1dc910594d0e4bd9a2dab655bf91032f36"}
Jul 08 23:07:54.774941 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.774832    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-115.ec2.internal" podStartSLOduration=21.774807375 podStartE2EDuration="21.774807375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:36.731237669 +0000 UTC m=+5.723070978" watchObservedRunningTime="2026-07-08 23:07:54.774807375 +0000 UTC m=+23.766640682"
Jul 08 23:07:54.793479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.793441    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-brmfm" podStartSLOduration=3.633485921 podStartE2EDuration="22.793429854s" totalImagesPullingTime="19.159943933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:54.792926119 +0000 UTC m=+23.784759425" watchObservedRunningTime="2026-07-08 23:07:54.793429854 +0000 UTC m=+23.785263160"
Jul 08 23:07:54.806756 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.806690    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zwhlx" podStartSLOduration=11.774339485 podStartE2EDuration="22.806677999s" totalImagesPullingTime="11.032338514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:54.806518043 +0000 UTC m=+23.798351360" watchObservedRunningTime="2026-07-08 23:07:54.806677999 +0000 UTC m=+23.798511305"
Jul 08 23:07:54.820845 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.820795    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hj86f" podStartSLOduration=11.763246952 podStartE2EDuration="22.820779902s" totalImagesPullingTime="11.05753295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:54.820579951 +0000 UTC m=+23.812413258" watchObservedRunningTime="2026-07-08 23:07:54.820779902 +0000 UTC m=+23.812613211"
Jul 08 23:07:54.833431 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.833398    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-45jkz" podStartSLOduration=11.764089877 podStartE2EDuration="22.833388453s" totalImagesPullingTime="11.069298576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:54.833287845 +0000 UTC m=+23.825121151" watchObservedRunningTime="2026-07-08 23:07:54.833388453 +0000 UTC m=+23.825221758"
Jul 08 23:07:54.849286 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:54.849249    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gzvzj" podStartSLOduration=3.651209761 podStartE2EDuration="22.849237768s" totalImagesPullingTime="19.198028007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:54.848874022 +0000 UTC m=+23.840707327" watchObservedRunningTime="2026-07-08 23:07:54.849237768 +0000 UTC m=+23.841071087"
Jul 08 23:07:55.431505 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:55.431486    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 23:07:55.514532 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:55.514507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:55.514645 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:55.514622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:55.765508 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:55.765427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4" event={"ID":"8eca8787-38ab-436c-9e9f-eeda050cf706","Type":"ContainerStarted","Data":"f27210f22306bb51e6564d1885f35b4f81dec45a0343ff8f474ace2cc304ba61"}
Jul 08 23:07:55.767055 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:55.767018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6v9qm" event={"ID":"f2522b38-49ba-4621-8a65-190d5a4f7ad2","Type":"ContainerStarted","Data":"cff50fecdcc2c4ba69997903439c6d318458474cd1de1722b77e752c4dba19ba"}
Jul 08 23:07:55.783650 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:55.783613    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6v9qm" podStartSLOduration=12.717694301 podStartE2EDuration="23.783596328s" totalImagesPullingTime="11.065902027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:55.783521974 +0000 UTC m=+24.775355280" watchObservedRunningTime="2026-07-08 23:07:55.783596328 +0000 UTC m=+24.775429636"
Jul 08 23:07:56.390907 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.390813    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T23:07:55.431502199Z","UUID":"bdf488b7-935b-4cec-9ff0-8c975f2ab0a1","Handler":null,"Name":"","Endpoint":""}
Jul 08 23:07:56.392346 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.392329    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 23:07:56.392460 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.392366    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 23:07:56.514925 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.514887    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:56.515086 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.514894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:56.515357 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:56.515035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:56.515459 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:56.515395    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:56.773882 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.773848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"e867fdc713abcc2923bfed309d8ad5d6e68c1c23cdb8e43525a3cadf7790efbb"}
Jul 08 23:07:56.776375 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:56.776341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4" event={"ID":"8eca8787-38ab-436c-9e9f-eeda050cf706","Type":"ContainerStarted","Data":"080707441de0781e039dce2cc13980ef9fde39d393bf1d16a0569dd603b14b8f"}
Jul 08 23:07:57.514822 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:57.514794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:57.514993 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:57.514909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:57.794482 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:57.794384    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2s6d4" podStartSLOduration=12.317655521 podStartE2EDuration="25.794365191s" totalImagesPullingTime="13.47670967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:57.794219163 +0000 UTC m=+26.786052470" watchObservedRunningTime="2026-07-08 23:07:57.794365191 +0000 UTC m=+26.786198499"
Jul 08 23:07:58.514327 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:58.514171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:07:58.514480 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:58.514229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:07:58.514480 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:58.514401    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:07:58.514480 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:58.514459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:07:59.514884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:59.514852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:07:59.515697 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:07:59.514970    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:07:59.541612 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:59.541578    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:59.542498 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:59.542475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:07:59.787168 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:59.787071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" event={"ID":"c5feb9d3-7783-4adf-add9-e666e310c820","Type":"ContainerStarted","Data":"65ea937d62ee85e10a63713504a5275f17e45955f741ea0c3eab2774f695cfc0"}
Jul 08 23:07:59.812536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:07:59.812494    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc" podStartSLOduration=8.60232799 podStartE2EDuration="27.812478054s" totalImagesPullingTime="19.210150064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:07:59.812415401 +0000 UTC m=+28.804248707" watchObservedRunningTime="2026-07-08 23:07:59.812478054 +0000 UTC m=+28.804311376"
Jul 08 23:08:00.514289 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.514101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:00.514444 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.514150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:00.514444 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:00.514382    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:08:00.514561 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:00.514501    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:08:00.789213 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.789131    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:08:00.789546 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.789215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:08:00.789546 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.789269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:08:00.804079 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.804059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:08:00.804232 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.804219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:08:00.846914 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.846888    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ftnp8"]
Jul 08 23:08:00.847019 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.846993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:00.847134 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:00.847110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:08:00.847530 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.847512    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4psn"]
Jul 08 23:08:00.847624 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.847601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:00.847761 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:00.847693    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:08:00.849299 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.849279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8dfb9"]
Jul 08 23:08:00.849391 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:00.849359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:00.849455 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:00.849440    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:08:01.792752 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:01.792719    2577 generic.go:356] "Generic (PLEG): container finished" podID="753217f0-e74b-40cf-a22f-6ddb1e4af404" containerID="ba9644953ca06bcfccc4ad8fae6f06888e8717d60b8bc2ff67d578b0a6645e90" exitCode=0
Jul 08 23:08:01.793207 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:01.792796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerDied","Data":"ba9644953ca06bcfccc4ad8fae6f06888e8717d60b8bc2ff67d578b0a6645e90"}
Jul 08 23:08:02.180223 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:02.180199    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:08:02.180364 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:02.180302    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 08 23:08:02.180720 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:02.180688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zwhlx"
Jul 08 23:08:02.514814 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:02.514723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:02.514965 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:02.514722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:02.514965 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:02.514839    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:08:02.514965 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:02.514946    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:08:02.515108 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:02.514979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:02.515108 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:02.515035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:08:03.800032 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:03.799806    2577 generic.go:356] "Generic (PLEG): container finished" podID="753217f0-e74b-40cf-a22f-6ddb1e4af404" containerID="feb974e7b4642019f7c9490c3099a5813e4cd5cb298589ee9a6af453df95cc72" exitCode=0
Jul 08 23:08:03.800032 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:03.799891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerDied","Data":"feb974e7b4642019f7c9490c3099a5813e4cd5cb298589ee9a6af453df95cc72"}
Jul 08 23:08:04.514405 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:04.514380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:04.514528 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:04.514394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:04.514528 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:04.514484    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:04.514528 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:04.514480    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:08:04.514679 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:04.514551    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:08:04.514679 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:04.514647    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:08:04.804865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:04.804790    2577 generic.go:356] "Generic (PLEG): container finished" podID="753217f0-e74b-40cf-a22f-6ddb1e4af404" containerID="cb25f1ef93dbe4340e3ab9c713dd15db17e190a29231b52f11c34d0fa1afa440" exitCode=0
Jul 08 23:08:04.804865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:04.804840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerDied","Data":"cb25f1ef93dbe4340e3ab9c713dd15db17e190a29231b52f11c34d0fa1afa440"}
Jul 08 23:08:05.099953 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:05.099886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:05.100093 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.100016    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:08:05.100093 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.100071    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:37.100054457 +0000 UTC m=+66.091887743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 23:08:05.200899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:05.200872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:05.201062 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.200987    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 23:08:05.201062 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.201035    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret podName:dbf63736-facd-4dd4-ac3e-418bc5356176 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:37.201019968 +0000 UTC m=+66.192853256 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret") pod "global-pull-secret-syncer-x4psn" (UID: "dbf63736-facd-4dd4-ac3e-418bc5356176") : object "kube-system"/"original-pull-secret" not registered
Jul 08 23:08:05.301660 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:05.301636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:05.301767 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.301751    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 23:08:05.301767 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.301764    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 23:08:05.301834 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.301772    2577 projected.go:196] Error preparing data for projected volume kube-api-access-p8fls for pod openshift-network-diagnostics/network-check-target-8dfb9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:08:05.301834 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:05.301810    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls podName:8e891f7f-acd5-4bd9-af25-8d247db98d02 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:37.301801031 +0000 UTC m=+66.293634315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p8fls" (UniqueName: "kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls") pod "network-check-target-8dfb9" (UID: "8e891f7f-acd5-4bd9-af25-8d247db98d02") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 23:08:06.514125 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.514093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:06.514605 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.514093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:06.514605 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:06.514213    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4psn" podUID="dbf63736-facd-4dd4-ac3e-418bc5356176"
Jul 08 23:08:06.514605 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.514093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:06.514605 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:06.514261    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8dfb9" podUID="8e891f7f-acd5-4bd9-af25-8d247db98d02"
Jul 08 23:08:06.514605 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:06.514334    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:08:06.780686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.780624    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-115.ec2.internal" event="NodeReady"
Jul 08 23:08:06.780833 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.780736    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 23:08:06.823852 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.823830    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7r4qc"]
Jul 08 23:08:06.827633 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.827618    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xzsqd"]
Jul 08 23:08:06.827767 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.827749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:06.830138 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.830113    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 23:08:06.830138 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.830135    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vhnmb\""
Jul 08 23:08:06.830275 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.830260    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 23:08:06.831722 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.831690    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:06.834009 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.833987    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtwjf\""
Jul 08 23:08:06.834109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.834049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 23:08:06.834341 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.834325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 23:08:06.834341 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.834337    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 23:08:06.834840 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.834744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xzsqd"]
Jul 08 23:08:06.837995 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.837971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7r4qc"]
Jul 08 23:08:06.912436 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.912412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-config-volume\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:06.912527 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.912449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65zv\" (UniqueName: \"kubernetes.io/projected/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-kube-api-access-t65zv\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:06.912527 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.912473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01ba5fc2-4c03-47c3-9bb6-54682cd7d25f-cert\") pod \"ingress-canary-xzsqd\" (UID: \"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f\") " pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:06.912613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.912544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-tmp-dir\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:06.912613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.912569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttkfw\" (UniqueName: \"kubernetes.io/projected/01ba5fc2-4c03-47c3-9bb6-54682cd7d25f-kube-api-access-ttkfw\") pod \"ingress-canary-xzsqd\" (UID: \"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f\") " pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:06.912613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:06.912608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.013028 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65zv\" (UniqueName: \"kubernetes.io/projected/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-kube-api-access-t65zv\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.013128 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01ba5fc2-4c03-47c3-9bb6-54682cd7d25f-cert\") pod \"ingress-canary-xzsqd\" (UID: \"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f\") " pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:07.013128 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-tmp-dir\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.013128 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttkfw\" (UniqueName: \"kubernetes.io/projected/01ba5fc2-4c03-47c3-9bb6-54682cd7d25f-kube-api-access-ttkfw\") pod \"ingress-canary-xzsqd\" (UID: \"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f\") " pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:07.013279 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.013279 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-config-volume\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.013399 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:07.013296    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:07.013399 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:07.013346    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:07.513327139 +0000 UTC m=+36.505160436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:07.013524 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013427    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-tmp-dir\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.013759 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.013732    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-config-volume\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.017532 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.017509    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01ba5fc2-4c03-47c3-9bb6-54682cd7d25f-cert\") pod \"ingress-canary-xzsqd\" (UID: \"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f\") " pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:07.021720 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.021673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65zv\" (UniqueName: \"kubernetes.io/projected/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-kube-api-access-t65zv\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.021879 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.021837    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttkfw\" (UniqueName: \"kubernetes.io/projected/01ba5fc2-4c03-47c3-9bb6-54682cd7d25f-kube-api-access-ttkfw\") pod \"ingress-canary-xzsqd\" (UID: \"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f\") " pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:07.143428 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.143404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xzsqd"
Jul 08 23:08:07.289808 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.289781    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xzsqd"]
Jul 08 23:08:07.294400 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:08:07.294213    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ba5fc2_4c03_47c3_9bb6_54682cd7d25f.slice/crio-327ac267a620f6ee696c3a8ccd6bc3065dd42a3252c43335bbfe5d0db5cf1a6d WatchSource:0}: Error finding container 327ac267a620f6ee696c3a8ccd6bc3065dd42a3252c43335bbfe5d0db5cf1a6d: Status 404 returned error can't find the container with id 327ac267a620f6ee696c3a8ccd6bc3065dd42a3252c43335bbfe5d0db5cf1a6d
Jul 08 23:08:07.516650 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.516564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:07.517351 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:07.516725    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:07.517351 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:07.516790    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:08.516772455 +0000 UTC m=+37.508605739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:07.811448 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:07.811378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xzsqd" event={"ID":"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f","Type":"ContainerStarted","Data":"327ac267a620f6ee696c3a8ccd6bc3065dd42a3252c43335bbfe5d0db5cf1a6d"}
Jul 08 23:08:08.514938 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.514861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:08.514938 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.514884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:08.515159 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.515032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:08.517901 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.517845    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 23:08:08.518927 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.518905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2g9tf\""
Jul 08 23:08:08.519243 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.519077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwbh8\""
Jul 08 23:08:08.519243 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.519131    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 23:08:08.519243 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.519143    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 23:08:08.519243 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.519172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 23:08:08.524174 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:08.524104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:08.524286 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:08.524252    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:08.524356 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:08.524319    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:10.524299144 +0000 UTC m=+39.516132429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:10.540103 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:10.540063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:10.540433 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:10.540177    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:10.540433 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:10.540230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:14.540216373 +0000 UTC m=+43.532049656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:10.821012 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:10.820975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xzsqd" event={"ID":"01ba5fc2-4c03-47c3-9bb6-54682cd7d25f","Type":"ContainerStarted","Data":"a6fd5cc2d7f926f336e580355d17bf7aa0a61dce3fb34ab8cde8783be42f6920"}
Jul 08 23:08:10.826034 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:10.826006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerStarted","Data":"d9d79d16ae8d116b0bd89d052bec361372fc035de84f5e792be949d91136792f"}
Jul 08 23:08:10.836928 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:10.836878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xzsqd" podStartSLOduration=1.457227999 podStartE2EDuration="4.836860456s" totalImagesPullingTime="3.379632457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:08:10.836234429 +0000 UTC m=+39.828067736" watchObservedRunningTime="2026-07-08 23:08:10.836860456 +0000 UTC m=+39.828693763"
Jul 08 23:08:11.830982 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:11.830944    2577 generic.go:356] "Generic (PLEG): container finished" podID="753217f0-e74b-40cf-a22f-6ddb1e4af404" containerID="d9d79d16ae8d116b0bd89d052bec361372fc035de84f5e792be949d91136792f" exitCode=0
Jul 08 23:08:11.831487 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:11.831017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerDied","Data":"d9d79d16ae8d116b0bd89d052bec361372fc035de84f5e792be949d91136792f"}
Jul 08 23:08:12.836292 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:12.836112    2577 generic.go:356] "Generic (PLEG): container finished" podID="753217f0-e74b-40cf-a22f-6ddb1e4af404" containerID="d91ffef0e81f0eb2b0ff33c32eaafcd9f1cc368a6589d30836f4efed2183f1b5" exitCode=0
Jul 08 23:08:12.836292 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:12.836197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerDied","Data":"d91ffef0e81f0eb2b0ff33c32eaafcd9f1cc368a6589d30836f4efed2183f1b5"}
Jul 08 23:08:13.841471 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:13.841437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" event={"ID":"753217f0-e74b-40cf-a22f-6ddb1e4af404","Type":"ContainerStarted","Data":"fda5ba812edac6cb255bec0fbe4a553f465b3988d4aec9cf01ad64ed77145249"}
Jul 08 23:08:13.863974 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:13.863932    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vd9d9" podStartSLOduration=9.070614599 podStartE2EDuration="41.863918668s" totalImagesPullingTime="32.793304069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:08:13.862174658 +0000 UTC m=+42.854007992" watchObservedRunningTime="2026-07-08 23:08:13.863918668 +0000 UTC m=+42.855751971"
Jul 08 23:08:14.570451 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:14.570422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:14.570596 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:14.570552    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:14.570633 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:14.570615    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:22.570595403 +0000 UTC m=+51.562428687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:22.619748 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:22.619686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:22.620265 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:22.619843    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:22.620265 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:22.619927    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:08:38.619909894 +0000 UTC m=+67.611743178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:32.817034 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:32.817003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zftwc"
Jul 08 23:08:34.111025 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.110981    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"]
Jul 08 23:08:34.113686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.113664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.116686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.116657    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 08 23:08:34.116914 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.116775    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 08 23:08:34.116914 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.116903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 23:08:34.117900 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.117874    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 23:08:34.118043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.117874    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 08 23:08:34.118043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.117877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 08 23:08:34.118043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.117938    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 23:08:34.125134 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.125109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"]
Jul 08 23:08:34.296804 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.296777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15968806-1e24-4c52-a0c7-6dce958c9888-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.296953 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.296817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-ca\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.296953 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.296849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-hub\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.296953 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.296908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnh2\" (UniqueName: \"kubernetes.io/projected/15968806-1e24-4c52-a0c7-6dce958c9888-kube-api-access-2tnh2\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.297118 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.296956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.297118 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.297027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.397597 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.397564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.397744 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.397612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15968806-1e24-4c52-a0c7-6dce958c9888-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.397744 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.397648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-ca\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.397744 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.397684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-hub\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.397914 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.397758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnh2\" (UniqueName: \"kubernetes.io/projected/15968806-1e24-4c52-a0c7-6dce958c9888-kube-api-access-2tnh2\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.397914 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.397791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.398486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.398452    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15968806-1e24-4c52-a0c7-6dce958c9888-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.401864 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.401831    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-ca\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.401864 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.401845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-hub\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.402011 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.401926    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.402011 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.401991    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15968806-1e24-4c52-a0c7-6dce958c9888-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.406850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.406827    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnh2\" (UniqueName: \"kubernetes.io/projected/15968806-1e24-4c52-a0c7-6dce958c9888-kube-api-access-2tnh2\") pod \"cluster-proxy-proxy-agent-6bd87f5499-kp44l\" (UID: \"15968806-1e24-4c52-a0c7-6dce958c9888\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.435621 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.435596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:08:34.553692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.553661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"]
Jul 08 23:08:34.557481 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:08:34.557456    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15968806_1e24_4c52_a0c7_6dce958c9888.slice/crio-6070273eff1b0c99fef1632f22a2ab4b886c367a57c99afcd0f3217247feda80 WatchSource:0}: Error finding container 6070273eff1b0c99fef1632f22a2ab4b886c367a57c99afcd0f3217247feda80: Status 404 returned error can't find the container with id 6070273eff1b0c99fef1632f22a2ab4b886c367a57c99afcd0f3217247feda80
Jul 08 23:08:34.889627 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:34.889595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" event={"ID":"15968806-1e24-4c52-a0c7-6dce958c9888","Type":"ContainerStarted","Data":"6070273eff1b0c99fef1632f22a2ab4b886c367a57c99afcd0f3217247feda80"}
Jul 08 23:08:37.118093 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.118064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:08:37.121338 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.121318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 23:08:37.128383 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:37.128361    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 23:08:37.128506 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:37.128422    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:09:41.128405185 +0000 UTC m=+130.120238473 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : secret "metrics-daemon-secret" not found
Jul 08 23:08:37.219251 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.219226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:37.221976 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.221957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 23:08:37.232509 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.232491    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbf63736-facd-4dd4-ac3e-418bc5356176-original-pull-secret\") pod \"global-pull-secret-syncer-x4psn\" (UID: \"dbf63736-facd-4dd4-ac3e-418bc5356176\") " pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:37.320139 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.320112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:37.322989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.322967    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 23:08:37.333320 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.333303    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 23:08:37.337204 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.337179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4psn"
Jul 08 23:08:37.344344 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.344317    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fls\" (UniqueName: \"kubernetes.io/projected/8e891f7f-acd5-4bd9-af25-8d247db98d02-kube-api-access-p8fls\") pod \"network-check-target-8dfb9\" (UID: \"8e891f7f-acd5-4bd9-af25-8d247db98d02\") " pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:37.476981 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.476946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4psn"]
Jul 08 23:08:37.631147 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.631119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2g9tf\""
Jul 08 23:08:37.638999 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:37.638973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:38.567423 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:08:38.567387    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf63736_facd_4dd4_ac3e_418bc5356176.slice/crio-c830991e1a2c3e29c5d301c4ad4008fa63430613a7db60968e63e480d5d0773f WatchSource:0}: Error finding container c830991e1a2c3e29c5d301c4ad4008fa63430613a7db60968e63e480d5d0773f: Status 404 returned error can't find the container with id c830991e1a2c3e29c5d301c4ad4008fa63430613a7db60968e63e480d5d0773f
Jul 08 23:08:38.632405 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:38.632378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:08:38.632592 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:38.632571    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:08:38.632683 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:08:38.632671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:09:10.632647828 +0000 UTC m=+99.624481127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:08:38.678668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:38.678643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8dfb9"]
Jul 08 23:08:38.775774 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:08:38.775748    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e891f7f_acd5_4bd9_af25_8d247db98d02.slice/crio-731c6298d71bdb43ad5992b33d9eb26a2894c817a5f7bba6aab9803d937a3c40 WatchSource:0}: Error finding container 731c6298d71bdb43ad5992b33d9eb26a2894c817a5f7bba6aab9803d937a3c40: Status 404 returned error can't find the container with id 731c6298d71bdb43ad5992b33d9eb26a2894c817a5f7bba6aab9803d937a3c40
Jul 08 23:08:38.900757 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:38.900729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8dfb9" event={"ID":"8e891f7f-acd5-4bd9-af25-8d247db98d02","Type":"ContainerStarted","Data":"731c6298d71bdb43ad5992b33d9eb26a2894c817a5f7bba6aab9803d937a3c40"}
Jul 08 23:08:38.902178 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:38.902157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" event={"ID":"15968806-1e24-4c52-a0c7-6dce958c9888","Type":"ContainerStarted","Data":"81a1bc61a14761423a5a75eaa5a086360af01845fac2dc7d9b4ada83d01cefea"}
Jul 08 23:08:38.903139 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:38.903121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4psn" event={"ID":"dbf63736-facd-4dd4-ac3e-418bc5356176","Type":"ContainerStarted","Data":"c830991e1a2c3e29c5d301c4ad4008fa63430613a7db60968e63e480d5d0773f"}
Jul 08 23:08:39.909084 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:39.909027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" event={"ID":"15968806-1e24-4c52-a0c7-6dce958c9888","Type":"ContainerStarted","Data":"1ae885f6484ce99de0831234eb83cf3cbd6215235e8a86d33fb6fd1216451a99"}
Jul 08 23:08:39.909084 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:39.909080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" event={"ID":"15968806-1e24-4c52-a0c7-6dce958c9888","Type":"ContainerStarted","Data":"95ec8e934510ffeca1ab13786dde2d2ab0c22c7b625d4a23e8135d82bfaef3b1"}
Jul 08 23:08:39.927332 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:39.927280    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" podStartSLOduration=1.6859816319999998 podStartE2EDuration="5.927263445s" totalImagesPullingTime="4.241281813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:08:39.927113898 +0000 UTC m=+68.918947204" watchObservedRunningTime="2026-07-08 23:08:39.927263445 +0000 UTC m=+68.919096728"
Jul 08 23:08:43.922656 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:43.922623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4psn" event={"ID":"dbf63736-facd-4dd4-ac3e-418bc5356176","Type":"ContainerStarted","Data":"03a8256827a8749f38ea10874df237356fb117495e00d3ca20492a9f0ee0f3e1"}
Jul 08 23:08:43.924231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:43.924206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8dfb9" event={"ID":"8e891f7f-acd5-4bd9-af25-8d247db98d02","Type":"ContainerStarted","Data":"01bf7c7c9bf250e1732db75d27700c928edac3c311d95ce10a362f049fbd9441"}
Jul 08 23:08:43.924345 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:43.924322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:08:43.937240 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:43.937197    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x4psn" podStartSLOduration=66.981986431 podStartE2EDuration="1m11.937184955s" totalImagesPullingTime="4.955198524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:08:43.936238106 +0000 UTC m=+72.928071413" watchObservedRunningTime="2026-07-08 23:08:43.937184955 +0000 UTC m=+72.929018261"
Jul 08 23:08:43.949551 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:08:43.949514    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8dfb9" podStartSLOduration=67.213518917 podStartE2EDuration="1m11.949502537s" totalImagesPullingTime="4.73598362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:08:43.949032646 +0000 UTC m=+72.940865951" watchObservedRunningTime="2026-07-08 23:08:43.949502537 +0000 UTC m=+72.941335843"
Jul 08 23:09:10.640765 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:09:10.640667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:09:10.641099 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:09:10.640793    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:09:10.641099 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:09:10.640874    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:10:14.640857586 +0000 UTC m=+163.632690870 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:09:14.930029 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:09:14.930001    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8dfb9"
Jul 08 23:09:41.145355 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:09:41.145309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:09:41.145859 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:09:41.145441    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 23:09:41.145859 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:09:41.145533    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:11:43.145511545 +0000 UTC m=+252.137344830 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : secret "metrics-daemon-secret" not found
Jul 08 23:10:09.837800 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:10:09.837747    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7r4qc" podUID="32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0"
Jul 08 23:10:10.171054 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:10:10.171026    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:10:11.543151 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:10:11.543116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:10:14.656750 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:10:14.656688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:10:14.657278 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:10:14.656851    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 23:10:14.657278 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:10:14.656990    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls podName:32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:16.656965561 +0000 UTC m=+285.648798847 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls") pod "dns-default-7r4qc" (UID: "32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0") : secret "dns-default-metrics-tls" not found
Jul 08 23:10:26.514096 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:10:26.514011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:11:43.194482 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:43.194433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:11:43.195000 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:43.194583    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 23:11:43.195000 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:43.194652    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs podName:e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6 nodeName:}" failed. No retries permitted until 2026-07-08 23:13:45.194636137 +0000 UTC m=+374.186469421 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs") pod "network-metrics-daemon-ftnp8" (UID: "e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6") : secret "metrics-daemon-secret" not found
Jul 08 23:11:58.159957 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.159918    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m9wx5"]
Jul 08 23:11:58.162033 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.162005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.164469 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.164439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 23:11:58.164622 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.164437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 23:11:58.164846 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.164764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 23:11:58.165546 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.165526    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f5gd8\""
Jul 08 23:11:58.165652 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.165568    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 23:11:58.172007 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.171982    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m9wx5"]
Jul 08 23:11:58.287346 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.287314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75a531ef-9012-4bb3-ba32-089218afc067-crio-socket\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.287346 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.287349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75a531ef-9012-4bb3-ba32-089218afc067-data-volume\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.287500 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.287391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.287500 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.287411    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlxq2\" (UniqueName: \"kubernetes.io/projected/75a531ef-9012-4bb3-ba32-089218afc067-kube-api-access-hlxq2\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.287500 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.287430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388375 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlxq2\" (UniqueName: \"kubernetes.io/projected/75a531ef-9012-4bb3-ba32-089218afc067-kube-api-access-hlxq2\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75a531ef-9012-4bb3-ba32-089218afc067-crio-socket\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388536 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.388478    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 23:11:58.388536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75a531ef-9012-4bb3-ba32-089218afc067-data-volume\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388842 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.388543    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:11:58.888524133 +0000 UTC m=+267.880357431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : secret "insights-runtime-extractor-tls" not found
Jul 08 23:11:58.388842 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.388548    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 23:11:58.388842 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388567    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/75a531ef-9012-4bb3-ba32-089218afc067-crio-socket\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.388842 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.388614    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:11:58.888600556 +0000 UTC m=+267.880433840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : configmap "kube-rbac-proxy" not found
Jul 08 23:11:58.389021 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.388917    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/75a531ef-9012-4bb3-ba32-089218afc067-data-volume\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.396822 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.396801    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlxq2\" (UniqueName: \"kubernetes.io/projected/75a531ef-9012-4bb3-ba32-089218afc067-kube-api-access-hlxq2\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.892634 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.892604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.892790 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:58.892640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:58.892790 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.892721    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 23:11:58.892790 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.892761    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 23:11:58.892790 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.892774    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:11:59.892757735 +0000 UTC m=+268.884591020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : configmap "kube-rbac-proxy" not found
Jul 08 23:11:58.892941 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:58.892826    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:11:59.892811699 +0000 UTC m=+268.884644984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : secret "insights-runtime-extractor-tls" not found
Jul 08 23:11:59.900418 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:59.900383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:59.900418 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:59.900422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:11:59.900855 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:59.900536    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 23:11:59.900855 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:59.900562    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 23:11:59.900855 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:59.900609    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:01.900590824 +0000 UTC m=+270.892424109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : configmap "kube-rbac-proxy" not found
Jul 08 23:11:59.900855 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:11:59.900625    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:01.90061719 +0000 UTC m=+270.892450474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : secret "insights-runtime-extractor-tls" not found
Jul 08 23:11:59.951249 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:11:59.951228    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hj86f_2c5b15c7-69bb-496a-9080-bb4ac43ae4fb/dns-node-resolver/0.log"
Jul 08 23:12:01.151338 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:01.151309    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-45jkz_88b0968b-f35e-48d1-9e4a-62d157eda27b/node-ca/0.log"
Jul 08 23:12:01.913798 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:01.913755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:01.913798 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:01.913805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:01.914046 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:01.913891    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 23:12:01.914046 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:01.913906    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 23:12:01.914046 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:01.913944    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:05.913929486 +0000 UTC m=+274.905762769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : configmap "kube-rbac-proxy" not found
Jul 08 23:12:01.914046 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:01.913988    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:05.913951424 +0000 UTC m=+274.905784709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : secret "insights-runtime-extractor-tls" not found
Jul 08 23:12:02.150977 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:02.150937    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzsqd_01ba5fc2-4c03-47c3-9bb6-54682cd7d25f/serve-healthcheck-canary/0.log"
Jul 08 23:12:05.938161 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:05.938118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:05.938662 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:05.938196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:05.938662 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:05.938268    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 23:12:05.938662 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:05.938304    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 23:12:05.938662 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:05.938369    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:13.938343331 +0000 UTC m=+282.930176615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : configmap "kube-rbac-proxy" not found
Jul 08 23:12:05.938662 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:05.938397    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls podName:75a531ef-9012-4bb3-ba32-089218afc067 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:13.938386559 +0000 UTC m=+282.930219845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9wx5" (UID: "75a531ef-9012-4bb3-ba32-089218afc067") : secret "insights-runtime-extractor-tls" not found
Jul 08 23:12:13.172210 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:13.172159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7r4qc" podUID="32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0"
Jul 08 23:12:13.529439 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:13.529366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:12:13.990771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:13.990731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:13.990771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:13.990777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:13.991416 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:13.991386    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/75a531ef-9012-4bb3-ba32-089218afc067-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:13.993355 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:13.993326    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/75a531ef-9012-4bb3-ba32-089218afc067-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9wx5\" (UID: \"75a531ef-9012-4bb3-ba32-089218afc067\") " pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:14.072516 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:14.072490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m9wx5"
Jul 08 23:12:14.196108 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:14.196079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m9wx5"]
Jul 08 23:12:14.199133 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:14.199108    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a531ef_9012_4bb3_ba32_089218afc067.slice/crio-8d9d0d1ef19ca4191bf0dc0050a4840014983046e7c680edd6830152666cff12 WatchSource:0}: Error finding container 8d9d0d1ef19ca4191bf0dc0050a4840014983046e7c680edd6830152666cff12: Status 404 returned error can't find the container with id 8d9d0d1ef19ca4191bf0dc0050a4840014983046e7c680edd6830152666cff12
Jul 08 23:12:14.533210 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:14.533176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9wx5" event={"ID":"75a531ef-9012-4bb3-ba32-089218afc067","Type":"ContainerStarted","Data":"8d9d0d1ef19ca4191bf0dc0050a4840014983046e7c680edd6830152666cff12"}
Jul 08 23:12:16.707826 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:16.707794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:12:16.710242 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:16.710213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0-metrics-tls\") pod \"dns-default-7r4qc\" (UID: \"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0\") " pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:12:16.833035 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:16.833010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vhnmb\""
Jul 08 23:12:16.841004 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:16.840986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:12:17.884314 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:17.884291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7r4qc"]
Jul 08 23:12:17.886992 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:17.886968    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ac82dc_4d55_486f_a9f9_6ae2a2a62bf0.slice/crio-8fae3257dd558bb80c4e31ce731bb3c31fbd0156a9089f3a0a35a3f0b1c325a3 WatchSource:0}: Error finding container 8fae3257dd558bb80c4e31ce731bb3c31fbd0156a9089f3a0a35a3f0b1c325a3: Status 404 returned error can't find the container with id 8fae3257dd558bb80c4e31ce731bb3c31fbd0156a9089f3a0a35a3f0b1c325a3
Jul 08 23:12:18.547613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:18.547570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9wx5" event={"ID":"75a531ef-9012-4bb3-ba32-089218afc067","Type":"ContainerStarted","Data":"81f6e800c0ea106b44a10f90da668c6b673d02ee688138ac3cc249045be5109d"}
Jul 08 23:12:18.548822 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:18.548792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r4qc" event={"ID":"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0","Type":"ContainerStarted","Data":"8fae3257dd558bb80c4e31ce731bb3c31fbd0156a9089f3a0a35a3f0b1c325a3"}
Jul 08 23:12:19.554206 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:19.554137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9wx5" event={"ID":"75a531ef-9012-4bb3-ba32-089218afc067","Type":"ContainerStarted","Data":"d8a713dd4a3c32462ac3875aec5df2fafe461ef46b4b929a74f0c2ce7df9aac2"}
Jul 08 23:12:20.559146 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:20.559106    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r4qc" event={"ID":"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0","Type":"ContainerStarted","Data":"ffbb6cf061ab0ba638634804009aeb8fbcbf53c9b861b76f4a4c9542fcc5c9d0"}
Jul 08 23:12:20.559547 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:20.559168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7r4qc" event={"ID":"32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0","Type":"ContainerStarted","Data":"3b8719b845d4b988b788ca5a56bc3fed4c161facc971d69a5f2f9fbbbf40087f"}
Jul 08 23:12:20.559547 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:20.559246    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:12:20.574525 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:20.574476    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7r4qc" podStartSLOduration=252.791487024 podStartE2EDuration="4m14.574457515s" totalImagesPullingTime="1.782970491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:20.573303126 +0000 UTC m=+289.565136430" watchObservedRunningTime="2026-07-08 23:12:20.574457515 +0000 UTC m=+289.566290822"
Jul 08 23:12:22.080032 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.079966    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-lsd2g"]
Jul 08 23:12:22.082861 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.082842    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:22.085180 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.085158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b2mlc\""
Jul 08 23:12:22.085290 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.085215    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 23:12:22.085349 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.085297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 23:12:22.092465 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.092442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-lsd2g"]
Jul 08 23:12:22.116167 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.116146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74f86749cc-7qlw4"]
Jul 08 23:12:22.118849 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.118834    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.121105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.121087    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 23:12:22.121197 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.121156    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 23:12:22.121197 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.121158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 23:12:22.121422 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.121409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sq6cw\""
Jul 08 23:12:22.126605 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.126587    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 23:12:22.129597 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.129576    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f86749cc-7qlw4"]
Jul 08 23:12:22.146955 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.146933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl66p\" (UniqueName: \"kubernetes.io/projected/f3ad60a8-7f4d-46d0-ad44-49fd54bec888-kube-api-access-rl66p\") pod \"downloads-685bfb5f64-lsd2g\" (UID: \"f3ad60a8-7f4d-46d0-ad44-49fd54bec888\") " pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:22.247190 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl66p\" (UniqueName: \"kubernetes.io/projected/f3ad60a8-7f4d-46d0-ad44-49fd54bec888-kube-api-access-rl66p\") pod \"downloads-685bfb5f64-lsd2g\" (UID: \"f3ad60a8-7f4d-46d0-ad44-49fd54bec888\") " pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:22.247282 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8c8c860-b7bd-448e-9828-2f4c877269f1-registry-certificates\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247282 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8c8c860-b7bd-448e-9828-2f4c877269f1-image-registry-private-configuration\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247282 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-registry-tls\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247379 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8c8c860-b7bd-448e-9828-2f4c877269f1-trusted-ca\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247379 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-bound-sa-token\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247379 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlzg\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-kube-api-access-jxlzg\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247379 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8c8c860-b7bd-448e-9828-2f4c877269f1-ca-trust-extracted\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.247505 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.247397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8c8c860-b7bd-448e-9828-2f4c877269f1-installation-pull-secrets\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.256624 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.256600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl66p\" (UniqueName: \"kubernetes.io/projected/f3ad60a8-7f4d-46d0-ad44-49fd54bec888-kube-api-access-rl66p\") pod \"downloads-685bfb5f64-lsd2g\" (UID: \"f3ad60a8-7f4d-46d0-ad44-49fd54bec888\") " pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:22.347772 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347714    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8c8c860-b7bd-448e-9828-2f4c877269f1-image-registry-private-configuration\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.347772 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-registry-tls\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.347890 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8c8c860-b7bd-448e-9828-2f4c877269f1-trusted-ca\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.347890 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347804    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-bound-sa-token\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.347890 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlzg\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-kube-api-access-jxlzg\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.347890 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8c8c860-b7bd-448e-9828-2f4c877269f1-ca-trust-extracted\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.348082 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8c8c860-b7bd-448e-9828-2f4c877269f1-installation-pull-secrets\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.348082 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.347940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8c8c860-b7bd-448e-9828-2f4c877269f1-registry-certificates\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.348383 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.348360    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b8c8c860-b7bd-448e-9828-2f4c877269f1-ca-trust-extracted\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.348819 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.348794    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b8c8c860-b7bd-448e-9828-2f4c877269f1-registry-certificates\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.349075 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.349056    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b8c8c860-b7bd-448e-9828-2f4c877269f1-trusted-ca\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.350169 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.350151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b8c8c860-b7bd-448e-9828-2f4c877269f1-installation-pull-secrets\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.350261 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.350206    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b8c8c860-b7bd-448e-9828-2f4c877269f1-image-registry-private-configuration\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.350436 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.350416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-registry-tls\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.357882 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.357858    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlzg\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-kube-api-access-jxlzg\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.358069 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.358049    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8c8c860-b7bd-448e-9828-2f4c877269f1-bound-sa-token\") pod \"image-registry-74f86749cc-7qlw4\" (UID: \"b8c8c860-b7bd-448e-9828-2f4c877269f1\") " pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.391322 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.391304    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:22.427484 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.427294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:22.519266 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.519091    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-lsd2g"]
Jul 08 23:12:22.522558 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:22.522529    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ad60a8_7f4d_46d0_ad44_49fd54bec888.slice/crio-f3e808b23583a26ca67da054bfcdf51f54dbf838b33f5fab04add0120327ebf9 WatchSource:0}: Error finding container f3e808b23583a26ca67da054bfcdf51f54dbf838b33f5fab04add0120327ebf9: Status 404 returned error can't find the container with id f3e808b23583a26ca67da054bfcdf51f54dbf838b33f5fab04add0120327ebf9
Jul 08 23:12:22.563672 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.563648    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74f86749cc-7qlw4"]
Jul 08 23:12:22.565857 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:22.565835    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c8c860_b7bd_448e_9828_2f4c877269f1.slice/crio-7e75621b2d690d6ca920c538d03864da7ef26ff5ea254740aaa4711178517002 WatchSource:0}: Error finding container 7e75621b2d690d6ca920c538d03864da7ef26ff5ea254740aaa4711178517002: Status 404 returned error can't find the container with id 7e75621b2d690d6ca920c538d03864da7ef26ff5ea254740aaa4711178517002
Jul 08 23:12:22.566538 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.566512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-lsd2g" event={"ID":"f3ad60a8-7f4d-46d0-ad44-49fd54bec888","Type":"ContainerStarted","Data":"f3e808b23583a26ca67da054bfcdf51f54dbf838b33f5fab04add0120327ebf9"}
Jul 08 23:12:22.568588 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.568569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9wx5" event={"ID":"75a531ef-9012-4bb3-ba32-089218afc067","Type":"ContainerStarted","Data":"1c9d9bd9126d5b1a671bedbde19e517325a2780ed70ce5ccfd7394f2862e8895"}
Jul 08 23:12:22.588374 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:22.588331    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m9wx5" podStartSLOduration=17.153518688 podStartE2EDuration="24.588318143s" totalImagesPullingTime="7.434799455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:11:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:22.587457758 +0000 UTC m=+291.579291065" watchObservedRunningTime="2026-07-08 23:12:22.588318143 +0000 UTC m=+291.580151448"
Jul 08 23:12:23.575669 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:23.575630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4" event={"ID":"b8c8c860-b7bd-448e-9828-2f4c877269f1","Type":"ContainerStarted","Data":"6fa6454620b4ff18e1db67e02ff4af6a5ef4e6204720674c429df184409f04a3"}
Jul 08 23:12:23.576135 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:23.575682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4" event={"ID":"b8c8c860-b7bd-448e-9828-2f4c877269f1","Type":"ContainerStarted","Data":"7e75621b2d690d6ca920c538d03864da7ef26ff5ea254740aaa4711178517002"}
Jul 08 23:12:23.576135 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:23.575977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:23.595520 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:23.595359    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4" podStartSLOduration=1.595340494 podStartE2EDuration="1.595340494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:23.59482512 +0000 UTC m=+292.586658427" watchObservedRunningTime="2026-07-08 23:12:23.595340494 +0000 UTC m=+292.587173802"
Jul 08 23:12:29.515314 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:29.515267    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ftnp8" podUID="e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6"
Jul 08 23:12:30.565032 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:30.565004    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7r4qc"
Jul 08 23:12:31.334718 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.334672    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"]
Jul 08 23:12:31.337974 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.337949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.340194 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.340162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 08 23:12:31.341067 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.341040    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 08 23:12:31.341206 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.341118    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 23:12:31.342380 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.341697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 23:12:31.342380 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.342327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mjjnr\""
Jul 08 23:12:31.346913 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.345918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"]
Jul 08 23:12:31.346913 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.346212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 23:12:31.412276 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.412249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvthw\" (UniqueName: \"kubernetes.io/projected/b65036aa-8028-4d0a-a38d-1bff65c5e464-kube-api-access-dvthw\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.412415 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.412290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b65036aa-8028-4d0a-a38d-1bff65c5e464-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.412415 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.412335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.412415 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.412383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.513046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.512913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.513046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.512978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.513046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.513038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvthw\" (UniqueName: \"kubernetes.io/projected/b65036aa-8028-4d0a-a38d-1bff65c5e464-kube-api-access-dvthw\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.518749 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.513066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b65036aa-8028-4d0a-a38d-1bff65c5e464-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.518749 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:31.513131    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 08 23:12:31.518749 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:31.513194    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-tls podName:b65036aa-8028-4d0a-a38d-1bff65c5e464 nodeName:}" failed. No retries permitted until 2026-07-08 23:12:32.013174383 +0000 UTC m=+301.005007670 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-qrhhp" (UID: "b65036aa-8028-4d0a-a38d-1bff65c5e464") : secret "prometheus-operator-tls" not found
Jul 08 23:12:31.518749 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.513800    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b65036aa-8028-4d0a-a38d-1bff65c5e464-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.518749 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.515689    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:31.521172 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:31.521151    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvthw\" (UniqueName: \"kubernetes.io/projected/b65036aa-8028-4d0a-a38d-1bff65c5e464-kube-api-access-dvthw\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:32.016827 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:32.016797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:32.019446 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:32.019421    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b65036aa-8028-4d0a-a38d-1bff65c5e464-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-qrhhp\" (UID: \"b65036aa-8028-4d0a-a38d-1bff65c5e464\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:32.253372 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:32.253339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"
Jul 08 23:12:38.389775 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:38.389751    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 23:12:38.481775 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:38.481738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp"]
Jul 08 23:12:38.485240 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:38.485217    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65036aa_8028_4d0a_a38d_1bff65c5e464.slice/crio-b1ea09a0bf7d467c3db037dabb98c8851fffa32ad4636aebb075e9d075c750e1 WatchSource:0}: Error finding container b1ea09a0bf7d467c3db037dabb98c8851fffa32ad4636aebb075e9d075c750e1: Status 404 returned error can't find the container with id b1ea09a0bf7d467c3db037dabb98c8851fffa32ad4636aebb075e9d075c750e1
Jul 08 23:12:38.487725 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:38.487694    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 23:12:38.626089 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:38.626028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp" event={"ID":"b65036aa-8028-4d0a-a38d-1bff65c5e464","Type":"ContainerStarted","Data":"b1ea09a0bf7d467c3db037dabb98c8851fffa32ad4636aebb075e9d075c750e1"}
Jul 08 23:12:39.632233 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:39.632168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-lsd2g" event={"ID":"f3ad60a8-7f4d-46d0-ad44-49fd54bec888","Type":"ContainerStarted","Data":"83d406c82dbf37d6f85c7e2b7fdab3e913b4ebf9e730f951eb0959ef4bd7ba23"}
Jul 08 23:12:39.632802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:39.632753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:39.646916 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:39.646892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-lsd2g"
Jul 08 23:12:39.652423 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:39.652373    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-lsd2g" podStartSLOduration=1.4506929 podStartE2EDuration="17.652355058s" totalImagesPullingTime="16.201662158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:39.650892102 +0000 UTC m=+308.642725409" watchObservedRunningTime="2026-07-08 23:12:39.652355058 +0000 UTC m=+308.644188364"
Jul 08 23:12:40.514078 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:40.514046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:12:40.638602 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:40.638567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp" event={"ID":"b65036aa-8028-4d0a-a38d-1bff65c5e464","Type":"ContainerStarted","Data":"f11858c35fdbc9f7704bd8c129f7cc9813c4a5fb4b6128cea5a84350d444a18a"}
Jul 08 23:12:40.639096 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:40.638631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp" event={"ID":"b65036aa-8028-4d0a-a38d-1bff65c5e464","Type":"ContainerStarted","Data":"d38f88845dc9fb6e1edab628acad4130b55a07ed217bc3e313f2f1de8e6e1172"}
Jul 08 23:12:40.656516 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:40.656429    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-qrhhp" podStartSLOduration=7.818369365 podStartE2EDuration="9.656410363s" totalImagesPullingTime="1.838040998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:40.656108078 +0000 UTC m=+309.647941387" watchObservedRunningTime="2026-07-08 23:12:40.656410363 +0000 UTC m=+309.648243669"
Jul 08 23:12:42.663920 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.663884    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"]
Jul 08 23:12:42.691420 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.691396    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"]
Jul 08 23:12:42.691573 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.691440    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b5zbg"]
Jul 08 23:12:42.691643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.691582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.693892 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.693866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 08 23:12:42.694031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.693895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v427l\""
Jul 08 23:12:42.694031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.693970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 08 23:12:42.713608 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.713583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.716053 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.716033    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 23:12:42.716156 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.716069    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q8zt5\""
Jul 08 23:12:42.716285 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.716267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 23:12:42.716358 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.716291    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 23:12:42.805680 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-textfile\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.805825 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45751c2-6054-46bc-8966-ddf9839896f7-metrics-client-ca\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.805825 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-sys\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.805825 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvst\" (UniqueName: \"kubernetes.io/projected/f45751c2-6054-46bc-8966-ddf9839896f7-kube-api-access-qjvst\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.805825 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0607942-de2c-40d5-ac35-65d7e30a200e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.806031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0607942-de2c-40d5-ac35-65d7e30a200e-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.806031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.806031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78cp\" (UniqueName: \"kubernetes.io/projected/c0607942-de2c-40d5-ac35-65d7e30a200e-kube-api-access-s78cp\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.806031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.805957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0607942-de2c-40d5-ac35-65d7e30a200e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.806031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.806011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-wtmp\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.806251 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.806041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-root\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.806251 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.806093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.806251 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.806132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-tls\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.906695 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-textfile\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.906832 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45751c2-6054-46bc-8966-ddf9839896f7-metrics-client-ca\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.906832 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-sys\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.906936 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvst\" (UniqueName: \"kubernetes.io/projected/f45751c2-6054-46bc-8966-ddf9839896f7-kube-api-access-qjvst\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.906936 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0607942-de2c-40d5-ac35-65d7e30a200e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.906936 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0607942-de2c-40d5-ac35-65d7e30a200e-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.907094 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-sys\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907094 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.906950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907094 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78cp\" (UniqueName: \"kubernetes.io/projected/c0607942-de2c-40d5-ac35-65d7e30a200e-kube-api-access-s78cp\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.907094 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907045    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-textfile\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907094 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0607942-de2c-40d5-ac35-65d7e30a200e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.907334 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-wtmp\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907334 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-root\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907334 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907334 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-tls\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907334 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907272    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-wtmp\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907334 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f45751c2-6054-46bc-8966-ddf9839896f7-root\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907619 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907359    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f45751c2-6054-46bc-8966-ddf9839896f7-metrics-client-ca\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.907675 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0607942-de2c-40d5-ac35-65d7e30a200e-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.907860 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.907814    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.909987 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.909954    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-tls\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.910235 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.910213    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0607942-de2c-40d5-ac35-65d7e30a200e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.910235 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.910226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0607942-de2c-40d5-ac35-65d7e30a200e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:42.910507 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.910484    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f45751c2-6054-46bc-8966-ddf9839896f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.918211 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.918114    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvst\" (UniqueName: \"kubernetes.io/projected/f45751c2-6054-46bc-8966-ddf9839896f7-kube-api-access-qjvst\") pod \"node-exporter-b5zbg\" (UID: \"f45751c2-6054-46bc-8966-ddf9839896f7\") " pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:42.918302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:42.918282    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78cp\" (UniqueName: \"kubernetes.io/projected/c0607942-de2c-40d5-ac35-65d7e30a200e-kube-api-access-s78cp\") pod \"openshift-state-metrics-54cf96dfcf-6rvtp\" (UID: \"c0607942-de2c-40d5-ac35-65d7e30a200e\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:43.001977 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.001955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"
Jul 08 23:12:43.023275 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.023255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b5zbg"
Jul 08 23:12:43.061251 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:43.061221    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45751c2_6054_46bc_8966_ddf9839896f7.slice/crio-f6e14a46164928322ccede319919373e647667ec1e8ce44617bbbd5e9fbd8258 WatchSource:0}: Error finding container f6e14a46164928322ccede319919373e647667ec1e8ce44617bbbd5e9fbd8258: Status 404 returned error can't find the container with id f6e14a46164928322ccede319919373e647667ec1e8ce44617bbbd5e9fbd8258
Jul 08 23:12:43.171424 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.171339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp"]
Jul 08 23:12:43.174390 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:43.174354    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0607942_de2c_40d5_ac35_65d7e30a200e.slice/crio-f68d9824cbfde9e130064643abe8594f07257d913bca6b64729743c79cae9eb3 WatchSource:0}: Error finding container f68d9824cbfde9e130064643abe8594f07257d913bca6b64729743c79cae9eb3: Status 404 returned error can't find the container with id f68d9824cbfde9e130064643abe8594f07257d913bca6b64729743c79cae9eb3
Jul 08 23:12:43.651214 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.651180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5zbg" event={"ID":"f45751c2-6054-46bc-8966-ddf9839896f7","Type":"ContainerStarted","Data":"f6e14a46164928322ccede319919373e647667ec1e8ce44617bbbd5e9fbd8258"}
Jul 08 23:12:43.653255 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.653224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp" event={"ID":"c0607942-de2c-40d5-ac35-65d7e30a200e","Type":"ContainerStarted","Data":"892b857707730b79c11b59835d67542ede9ca4c5663353b0eacb5b2cf0386fe8"}
Jul 08 23:12:43.653392 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.653285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp" event={"ID":"c0607942-de2c-40d5-ac35-65d7e30a200e","Type":"ContainerStarted","Data":"60a4dd0761c2cc7dd20949fd78dd35d5c57d16d8fed8edc9253fc55e8b0feeb2"}
Jul 08 23:12:43.653392 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.653312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp" event={"ID":"c0607942-de2c-40d5-ac35-65d7e30a200e","Type":"ContainerStarted","Data":"f68d9824cbfde9e130064643abe8594f07257d913bca6b64729743c79cae9eb3"}
Jul 08 23:12:43.749995 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.749520    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 23:12:43.768817 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.767856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 23:12:43.768817 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.768006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.771112 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.770935    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 08 23:12:43.771112 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.771033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 08 23:12:43.771112 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.770937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 08 23:12:43.771487 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.771412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 08 23:12:43.771734 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.771717    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 08 23:12:43.771823 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.771742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 08 23:12:43.772012 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.771994    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 08 23:12:43.772083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.772056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 08 23:12:43.772155 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.771996    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 08 23:12:43.772542 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.772524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sktm9\""
Jul 08 23:12:43.918002 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.917968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/989e9857-e6b5-4ca9-a2af-721b899994ae-config-out\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918184 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr48q\" (UniqueName: \"kubernetes.io/projected/989e9857-e6b5-4ca9-a2af-721b899994ae-kube-api-access-fr48q\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918184 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918184 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/989e9857-e6b5-4ca9-a2af-721b899994ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918184 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918184 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-web-config\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/989e9857-e6b5-4ca9-a2af-721b899994ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/989e9857-e6b5-4ca9-a2af-721b899994ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:43.918504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:43.918398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989e9857-e6b5-4ca9-a2af-721b899994ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.018954 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.018883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/989e9857-e6b5-4ca9-a2af-721b899994ae-config-out\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.018954 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.018925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr48q\" (UniqueName: \"kubernetes.io/projected/989e9857-e6b5-4ca9-a2af-721b899994ae-kube-api-access-fr48q\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.018954 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.018953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019208 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.018992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/989e9857-e6b5-4ca9-a2af-721b899994ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019208 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019208 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:44.019101    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 08 23:12:44.019208 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-web-config\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019208 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:44.019177    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls podName:989e9857-e6b5-4ca9-a2af-721b899994ae nodeName:}" failed. No retries permitted until 2026-07-08 23:12:44.519157887 +0000 UTC m=+313.510991177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "989e9857-e6b5-4ca9-a2af-721b899994ae") : secret "alertmanager-main-tls" not found
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/989e9857-e6b5-4ca9-a2af-721b899994ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/989e9857-e6b5-4ca9-a2af-721b899994ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/989e9857-e6b5-4ca9-a2af-721b899994ae-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.019817 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.019478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989e9857-e6b5-4ca9-a2af-721b899994ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.020866 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.020606    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/989e9857-e6b5-4ca9-a2af-721b899994ae-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.021212 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.021161    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/989e9857-e6b5-4ca9-a2af-721b899994ae-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.022230 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.022039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-web-config\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.023267 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.023241    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-config-volume\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.023373 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.023337    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/989e9857-e6b5-4ca9-a2af-721b899994ae-config-out\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.023734 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.023671    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/989e9857-e6b5-4ca9-a2af-721b899994ae-tls-assets\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.023874 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.023852    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.024141 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.024092    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.024792 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.024770    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.024883 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.024787    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.028038 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.028018    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr48q\" (UniqueName: \"kubernetes.io/projected/989e9857-e6b5-4ca9-a2af-721b899994ae-kube-api-access-fr48q\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.523284 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.523245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:44.523448 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:44.523405    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 08 23:12:44.523512 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:12:44.523488    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls podName:989e9857-e6b5-4ca9-a2af-721b899994ae nodeName:}" failed. No retries permitted until 2026-07-08 23:12:45.523466727 +0000 UTC m=+314.515300014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "989e9857-e6b5-4ca9-a2af-721b899994ae") : secret "alertmanager-main-tls" not found
Jul 08 23:12:44.583332 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:44.583303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74f86749cc-7qlw4"
Jul 08 23:12:45.530921 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.530897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:45.533498 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.533471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/989e9857-e6b5-4ca9-a2af-721b899994ae-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"989e9857-e6b5-4ca9-a2af-721b899994ae\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:45.581601 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.581573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 23:12:45.669719 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.669679    2577 generic.go:356] "Generic (PLEG): container finished" podID="f45751c2-6054-46bc-8966-ddf9839896f7" containerID="a50d5eab26861c4582ac4f5c490082f7e3cdc177505f95480321c7dc3c2d6a7b" exitCode=0
Jul 08 23:12:45.669857 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.669807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5zbg" event={"ID":"f45751c2-6054-46bc-8966-ddf9839896f7","Type":"ContainerDied","Data":"a50d5eab26861c4582ac4f5c490082f7e3cdc177505f95480321c7dc3c2d6a7b"}
Jul 08 23:12:45.672611 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.672574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp" event={"ID":"c0607942-de2c-40d5-ac35-65d7e30a200e","Type":"ContainerStarted","Data":"e94ea15ce8bdca700d778b17aaf0b10c89b4659742a89963c50a7def26869a0e"}
Jul 08 23:12:45.709098 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.709044    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-6rvtp" podStartSLOduration=1.755981652 podStartE2EDuration="3.709026553s" totalImagesPullingTime="1.953044901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:45.707803834 +0000 UTC m=+314.699637143" watchObservedRunningTime="2026-07-08 23:12:45.709026553 +0000 UTC m=+314.700859836"
Jul 08 23:12:45.746224 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:45.746117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 23:12:45.748062 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:12:45.748038    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989e9857_e6b5_4ca9_a2af_721b899994ae.slice/crio-9c20e4bd3164d5124060a9297835800b19865513cc3b186e8cdf60c0ae24ac68 WatchSource:0}: Error finding container 9c20e4bd3164d5124060a9297835800b19865513cc3b186e8cdf60c0ae24ac68: Status 404 returned error can't find the container with id 9c20e4bd3164d5124060a9297835800b19865513cc3b186e8cdf60c0ae24ac68
Jul 08 23:12:46.679089 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:46.679051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"9c20e4bd3164d5124060a9297835800b19865513cc3b186e8cdf60c0ae24ac68"}
Jul 08 23:12:46.683337 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:46.683306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5zbg" event={"ID":"f45751c2-6054-46bc-8966-ddf9839896f7","Type":"ContainerStarted","Data":"49e00eac82cf54dd17308330fdbcaae6238b51d4b0165fff311962ad596a6b82"}
Jul 08 23:12:46.683451 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:46.683362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5zbg" event={"ID":"f45751c2-6054-46bc-8966-ddf9839896f7","Type":"ContainerStarted","Data":"a3c0f61c61e7bce6bab9e154766c2dbb4692bf531e568091340833108283f80e"}
Jul 08 23:12:46.705791 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:46.705742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b5zbg" podStartSLOduration=2.589091033 podStartE2EDuration="4.705725473s" totalImagesPullingTime="2.030578757s" totalInitContainerRuntime="86.055683ms" isStatefulPod=true podCreationTimestamp="2026-07-08 23:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:46.70445794 +0000 UTC m=+315.696291256" watchObservedRunningTime="2026-07-08 23:12:46.705725473 +0000 UTC m=+315.697558773"
Jul 08 23:12:47.688424 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:47.688399    2577 generic.go:356] "Generic (PLEG): container finished" podID="989e9857-e6b5-4ca9-a2af-721b899994ae" containerID="94861f75ed22b2e97cb12a68d976010169ff9cb14ed8b94e02ba9d7133a415e7" exitCode=0
Jul 08 23:12:47.688827 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:47.688484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerDied","Data":"94861f75ed22b2e97cb12a68d976010169ff9cb14ed8b94e02ba9d7133a415e7"}
Jul 08 23:12:50.711535 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:50.711511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"19072bd7149d8ff9de933fd01031b3abb8412c54e1a9565981ae67a672e08622"}
Jul 08 23:12:50.711884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:50.711570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"2592163cb0732ade6d97ecd27810c7224cf8f14dcd437876f5071b3595efbb85"}
Jul 08 23:12:50.711884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:50.711598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"4e153bc963fbf9b58ffaf9cb9b2f5861eccaf290fd63408eca2b44ca4bae01aa"}
Jul 08 23:12:50.711884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:50.711621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"0b946b3dd13bd3e1db0d2cbbfcd5c08a4c85fd62d164f617d0fba376bd9e4426"}
Jul 08 23:12:50.711884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:50.711644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"3256949ae4dfd9736f09462df9e7cf577a7a02c63127e9185d597baef7dd24af"}
Jul 08 23:12:52.720986 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:52.720946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"989e9857-e6b5-4ca9-a2af-721b899994ae","Type":"ContainerStarted","Data":"79b9ff04cc6ab49c1330929fe12b313c3445b82b418aaa5eaa98957916b03b66"}
Jul 08 23:12:52.748117 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:12:52.748072    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.473583042 podStartE2EDuration="9.748057326s" totalImagesPullingTime="5.249501428s" totalInitContainerRuntime="24.972856ms" isStatefulPod=false podCreationTimestamp="2026-07-08 23:12:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:12:52.745472239 +0000 UTC m=+321.737305545" watchObservedRunningTime="2026-07-08 23:12:52.748057326 +0000 UTC m=+321.739890631"
Jul 08 23:13:04.437291 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:04.437245    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" podUID="15968806-1e24-4c52-a0c7-6dce958c9888" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 08 23:13:14.437437 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:14.437395    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" podUID="15968806-1e24-4c52-a0c7-6dce958c9888" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 08 23:13:24.437171 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.437074    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" podUID="15968806-1e24-4c52-a0c7-6dce958c9888" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 08 23:13:24.437567 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.437179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l"
Jul 08 23:13:24.437723 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.437666    2577 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1ae885f6484ce99de0831234eb83cf3cbd6215235e8a86d33fb6fd1216451a99"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 08 23:13:24.437781 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.437754    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" podUID="15968806-1e24-4c52-a0c7-6dce958c9888" containerName="service-proxy" containerID="cri-o://1ae885f6484ce99de0831234eb83cf3cbd6215235e8a86d33fb6fd1216451a99" gracePeriod=30
Jul 08 23:13:24.834751 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.834656    2577 generic.go:356] "Generic (PLEG): container finished" podID="15968806-1e24-4c52-a0c7-6dce958c9888" containerID="1ae885f6484ce99de0831234eb83cf3cbd6215235e8a86d33fb6fd1216451a99" exitCode=2
Jul 08 23:13:24.834751 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.834725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" event={"ID":"15968806-1e24-4c52-a0c7-6dce958c9888","Type":"ContainerDied","Data":"1ae885f6484ce99de0831234eb83cf3cbd6215235e8a86d33fb6fd1216451a99"}
Jul 08 23:13:24.834918 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:24.834761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bd87f5499-kp44l" event={"ID":"15968806-1e24-4c52-a0c7-6dce958c9888","Type":"ContainerStarted","Data":"2ca06186e8bad1d1bb1ec04c831ed961856a2d53a7bdfb504f25d92ce63f9e47"}
Jul 08 23:13:45.241958 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:45.241916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:13:45.244605 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:45.244576    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6-metrics-certs\") pod \"network-metrics-daemon-ftnp8\" (UID: \"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6\") " pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:13:45.317509 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:45.317481    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwbh8\""
Jul 08 23:13:45.325547 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:45.325528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ftnp8"
Jul 08 23:13:45.453317 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:45.453287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ftnp8"]
Jul 08 23:13:45.456064 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:13:45.456034    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90fa5b1_0bc9_4715_8f33_2f9eb8c421c6.slice/crio-ce90cfa64751157664b7f2b7451fd1d7e1eea8b0619bb2805fdc434ded856972 WatchSource:0}: Error finding container ce90cfa64751157664b7f2b7451fd1d7e1eea8b0619bb2805fdc434ded856972: Status 404 returned error can't find the container with id ce90cfa64751157664b7f2b7451fd1d7e1eea8b0619bb2805fdc434ded856972
Jul 08 23:13:45.909101 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:45.909070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ftnp8" event={"ID":"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6","Type":"ContainerStarted","Data":"ce90cfa64751157664b7f2b7451fd1d7e1eea8b0619bb2805fdc434ded856972"}
Jul 08 23:13:46.915040 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:46.915002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ftnp8" event={"ID":"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6","Type":"ContainerStarted","Data":"0c9ed3df54b0a32ee9873820b98eaf0217019016ccb6fe5ff87420d0b09c2c51"}
Jul 08 23:13:46.915368 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:46.915063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ftnp8" event={"ID":"e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6","Type":"ContainerStarted","Data":"a36f4c7631fb7b05031e5754d4370186ef191f9cc093a0909f718aa1fdc713b1"}
Jul 08 23:13:46.929926 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:13:46.929881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ftnp8" podStartSLOduration=373.863795793 podStartE2EDuration="6m14.929865733s" totalImagesPullingTime="1.06606994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:13:46.928624009 +0000 UTC m=+375.920457317" watchObservedRunningTime="2026-07-08 23:13:46.929865733 +0000 UTC m=+375.921699039"
Jul 08 23:15:46.534541 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.534505    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z6lg2"]
Jul 08 23:15:46.536945 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.536925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.539365 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.539338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 08 23:15:46.539504 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.539361    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5g7x6\""
Jul 08 23:15:46.540235 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.540215    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 08 23:15:46.545364 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.545008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z6lg2"]
Jul 08 23:15:46.657397 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.657370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af4037d-bae3-46b0-9c51-2b7a0747fe85-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z6lg2\" (UID: \"2af4037d-bae3-46b0-9c51-2b7a0747fe85\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.657511 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.657403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68zw\" (UniqueName: \"kubernetes.io/projected/2af4037d-bae3-46b0-9c51-2b7a0747fe85-kube-api-access-x68zw\") pod \"cert-manager-webhook-6486ffd997-z6lg2\" (UID: \"2af4037d-bae3-46b0-9c51-2b7a0747fe85\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.757985 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.757958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af4037d-bae3-46b0-9c51-2b7a0747fe85-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z6lg2\" (UID: \"2af4037d-bae3-46b0-9c51-2b7a0747fe85\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.758085 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.757989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68zw\" (UniqueName: \"kubernetes.io/projected/2af4037d-bae3-46b0-9c51-2b7a0747fe85-kube-api-access-x68zw\") pod \"cert-manager-webhook-6486ffd997-z6lg2\" (UID: \"2af4037d-bae3-46b0-9c51-2b7a0747fe85\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.766584 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.766556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68zw\" (UniqueName: \"kubernetes.io/projected/2af4037d-bae3-46b0-9c51-2b7a0747fe85-kube-api-access-x68zw\") pod \"cert-manager-webhook-6486ffd997-z6lg2\" (UID: \"2af4037d-bae3-46b0-9c51-2b7a0747fe85\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.766673 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.766628    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2af4037d-bae3-46b0-9c51-2b7a0747fe85-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z6lg2\" (UID: \"2af4037d-bae3-46b0-9c51-2b7a0747fe85\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.848378 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.848324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:46.975649 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:46.975627    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z6lg2"]
Jul 08 23:15:46.978151 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:15:46.978124    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2af4037d_bae3_46b0_9c51_2b7a0747fe85.slice/crio-04d821dc3ff36d12539caddb5d06121433aaf6a8ee91d776ab0e25503fd2066e WatchSource:0}: Error finding container 04d821dc3ff36d12539caddb5d06121433aaf6a8ee91d776ab0e25503fd2066e: Status 404 returned error can't find the container with id 04d821dc3ff36d12539caddb5d06121433aaf6a8ee91d776ab0e25503fd2066e
Jul 08 23:15:47.349389 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:47.349362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2" event={"ID":"2af4037d-bae3-46b0-9c51-2b7a0747fe85","Type":"ContainerStarted","Data":"04d821dc3ff36d12539caddb5d06121433aaf6a8ee91d776ab0e25503fd2066e"}
Jul 08 23:15:50.369443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:50.369414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2" event={"ID":"2af4037d-bae3-46b0-9c51-2b7a0747fe85","Type":"ContainerStarted","Data":"f0be2445a8a9679e5fa533e3ce5f00fddfaf92967e424e73a4fad4563c4ea6a4"}
Jul 08 23:15:50.369805 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:50.369482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:15:50.386254 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:50.386213    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2" podStartSLOduration=1.257905548 podStartE2EDuration="4.386198635s" totalImagesPullingTime="3.128293087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:15:50.385025415 +0000 UTC m=+499.376858720" watchObservedRunningTime="2026-07-08 23:15:50.386198635 +0000 UTC m=+499.378031942"
Jul 08 23:15:56.375608 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:15:56.375574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z6lg2"
Jul 08 23:16:15.280991 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.280960    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"]
Jul 08 23:16:15.287315 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.287298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.292337 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.292272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 08 23:16:15.292485 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.292401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k6v7k\""
Jul 08 23:16:15.292485 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.292307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 08 23:16:15.292938 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.292917    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 08 23:16:15.293046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.292919    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 08 23:16:15.301389 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.301366    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"]
Jul 08 23:16:15.351071 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.351046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwlj\" (UniqueName: \"kubernetes.io/projected/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-kube-api-access-zgwlj\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.351169 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.351090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-manifests\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.351169 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.351117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-tmp\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.351169 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.351133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.351169 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.351154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.451849 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.451820    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwlj\" (UniqueName: \"kubernetes.io/projected/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-kube-api-access-zgwlj\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.451947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.451869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-manifests\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.451947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.451886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-tmp\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.451947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.451905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.451947 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.451935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.452276 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.452255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-manifests\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.454215 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.454194    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-tmp\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.454443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.454425    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.454510 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.454473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.466797 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.466766    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwlj\" (UniqueName: \"kubernetes.io/projected/c35da1e3-ce78-42c2-b79f-aa7769c9aa30-kube-api-access-zgwlj\") pod \"opendatahub-operator-controller-manager-7b86895794-8hwqb\" (UID: \"c35da1e3-ce78-42c2-b79f-aa7769c9aa30\") " pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.596681 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.596627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:15.738761 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:15.738739    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"]
Jul 08 23:16:15.741312 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:16:15.741286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35da1e3_ce78_42c2_b79f_aa7769c9aa30.slice/crio-acba90fd9b334f0715f9148d071c7b7cfaca0e6570848d36a66a3863b918817e WatchSource:0}: Error finding container acba90fd9b334f0715f9148d071c7b7cfaca0e6570848d36a66a3863b918817e: Status 404 returned error can't find the container with id acba90fd9b334f0715f9148d071c7b7cfaca0e6570848d36a66a3863b918817e
Jul 08 23:16:16.466808 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:16.466764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb" event={"ID":"c35da1e3-ce78-42c2-b79f-aa7769c9aa30","Type":"ContainerStarted","Data":"acba90fd9b334f0715f9148d071c7b7cfaca0e6570848d36a66a3863b918817e"}
Jul 08 23:16:19.482593 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:19.482560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb" event={"ID":"c35da1e3-ce78-42c2-b79f-aa7769c9aa30","Type":"ContainerStarted","Data":"577c1682af9b090b6cfd84e7587e96ab38e21f9e066a738242c3c87dce086871"}
Jul 08 23:16:20.488452 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:20.488420    2577 generic.go:356] "Generic (PLEG): container finished" podID="c35da1e3-ce78-42c2-b79f-aa7769c9aa30" containerID="577c1682af9b090b6cfd84e7587e96ab38e21f9e066a738242c3c87dce086871" exitCode=0
Jul 08 23:16:20.488881 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:20.488513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb" event={"ID":"c35da1e3-ce78-42c2-b79f-aa7769c9aa30","Type":"ContainerDied","Data":"577c1682af9b090b6cfd84e7587e96ab38e21f9e066a738242c3c87dce086871"}
Jul 08 23:16:21.493407 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:21.493369    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb" event={"ID":"c35da1e3-ce78-42c2-b79f-aa7769c9aa30","Type":"ContainerStarted","Data":"46e85735f209c41306f778f27df739483bebbc1b5b01e57b221416034df2f935"}
Jul 08 23:16:21.493845 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:21.493538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:32.499951 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:32.499877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb"
Jul 08 23:16:32.538107 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:32.538046    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b86895794-8hwqb" podStartSLOduration=13.191922051 podStartE2EDuration="17.53803248s" totalImagesPullingTime="3.331730369s" totalInitContainerRuntime="1.01438006s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:16:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:16:21.526751016 +0000 UTC m=+530.518584321" watchObservedRunningTime="2026-07-08 23:16:32.53803248 +0000 UTC m=+541.529865785"
Jul 08 23:16:35.760917 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.760880    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-btndx"]
Jul 08 23:16:35.765962 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.765939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:35.771027 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.771003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 08 23:16:35.774886 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.774863    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-v4gp8\""
Jul 08 23:16:35.828029 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.828005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-btndx"]
Jul 08 23:16:35.897165 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.897140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rllwp\" (UniqueName: \"kubernetes.io/projected/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-kube-api-access-rllwp\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:35.897263 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.897167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-cert\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:35.997779 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.997754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rllwp\" (UniqueName: \"kubernetes.io/projected/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-kube-api-access-rllwp\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:35.997893 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:35.997784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-cert\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:35.997893 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:16:35.997883    2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 08 23:16:35.997968 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:16:35.997938    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-cert podName:20bfb057-f2e5-4ee8-a0eb-ec32f856c558 nodeName:}" failed. No retries permitted until 2026-07-08 23:16:36.497922182 +0000 UTC m=+545.489755466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-cert") pod "odh-model-controller-7998c67dc4-btndx" (UID: "20bfb057-f2e5-4ee8-a0eb-ec32f856c558") : secret "odh-model-controller-webhook-cert" not found
Jul 08 23:16:36.030885 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:36.030821    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rllwp\" (UniqueName: \"kubernetes.io/projected/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-kube-api-access-rllwp\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:36.501855 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:36.501827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-cert\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:36.504306 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:36.504285    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20bfb057-f2e5-4ee8-a0eb-ec32f856c558-cert\") pod \"odh-model-controller-7998c67dc4-btndx\" (UID: \"20bfb057-f2e5-4ee8-a0eb-ec32f856c558\") " pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:36.677348 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:36.677321    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:36.834849 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:36.834820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-btndx"]
Jul 08 23:16:36.838640 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:16:36.838609    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bfb057_f2e5_4ee8_a0eb_ec32f856c558.slice/crio-f3b552173fdb5d9f77ae39b8a9ee18981f00ac5b25293d94ae2bb6c6eac103fc WatchSource:0}: Error finding container f3b552173fdb5d9f77ae39b8a9ee18981f00ac5b25293d94ae2bb6c6eac103fc: Status 404 returned error can't find the container with id f3b552173fdb5d9f77ae39b8a9ee18981f00ac5b25293d94ae2bb6c6eac103fc
Jul 08 23:16:37.550522 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:37.550491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-btndx" event={"ID":"20bfb057-f2e5-4ee8-a0eb-ec32f856c558","Type":"ContainerStarted","Data":"f3b552173fdb5d9f77ae39b8a9ee18981f00ac5b25293d94ae2bb6c6eac103fc"}
Jul 08 23:16:38.120610 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.120575    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"]
Jul 08 23:16:38.123829 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.123801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.136668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.136640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 08 23:16:38.136809 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.136782    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 08 23:16:38.137916 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.137895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kcgmn\""
Jul 08 23:16:38.138020 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.137933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 23:16:38.138020 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.137936    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 23:16:38.182206 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.182185    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"]
Jul 08 23:16:38.214220 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.214195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ca08daf-44f5-406d-aa54-dc36507f0b47-tmp\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.214311 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.214241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca08daf-44f5-406d-aa54-dc36507f0b47-tls-certs\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.214311 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.214268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbbh\" (UniqueName: \"kubernetes.io/projected/1ca08daf-44f5-406d-aa54-dc36507f0b47-kube-api-access-7tbbh\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.315463 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.315430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca08daf-44f5-406d-aa54-dc36507f0b47-tls-certs\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.315592 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.315472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbbh\" (UniqueName: \"kubernetes.io/projected/1ca08daf-44f5-406d-aa54-dc36507f0b47-kube-api-access-7tbbh\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.315592 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.315567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ca08daf-44f5-406d-aa54-dc36507f0b47-tmp\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.315592 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:16:38.315578    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 08 23:16:38.315794 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:16:38.315638    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca08daf-44f5-406d-aa54-dc36507f0b47-tls-certs podName:1ca08daf-44f5-406d-aa54-dc36507f0b47 nodeName:}" failed. No retries permitted until 2026-07-08 23:16:38.815617521 +0000 UTC m=+547.807450820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/1ca08daf-44f5-406d-aa54-dc36507f0b47-tls-certs") pod "kube-auth-proxy-7b8998bd98-k9259" (UID: "1ca08daf-44f5-406d-aa54-dc36507f0b47") : secret "kube-auth-proxy-tls" not found
Jul 08 23:16:38.318115 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.318090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1ca08daf-44f5-406d-aa54-dc36507f0b47-tmp\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.351970 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.351939    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbbh\" (UniqueName: \"kubernetes.io/projected/1ca08daf-44f5-406d-aa54-dc36507f0b47-kube-api-access-7tbbh\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.819750 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.819717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca08daf-44f5-406d-aa54-dc36507f0b47-tls-certs\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:38.822020 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:38.822000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca08daf-44f5-406d-aa54-dc36507f0b47-tls-certs\") pod \"kube-auth-proxy-7b8998bd98-k9259\" (UID: \"1ca08daf-44f5-406d-aa54-dc36507f0b47\") " pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:39.036181 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:39.036149    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"
Jul 08 23:16:39.208215 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:39.208177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b8998bd98-k9259"]
Jul 08 23:16:40.268205 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:16:40.268169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca08daf_44f5_406d_aa54_dc36507f0b47.slice/crio-7676af9957b1ba461336ad5a1a45e9767b6ba58e932efed753106a0091e595da WatchSource:0}: Error finding container 7676af9957b1ba461336ad5a1a45e9767b6ba58e932efed753106a0091e595da: Status 404 returned error can't find the container with id 7676af9957b1ba461336ad5a1a45e9767b6ba58e932efed753106a0091e595da
Jul 08 23:16:40.564362 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:40.564287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259" event={"ID":"1ca08daf-44f5-406d-aa54-dc36507f0b47","Type":"ContainerStarted","Data":"7676af9957b1ba461336ad5a1a45e9767b6ba58e932efed753106a0091e595da"}
Jul 08 23:16:40.565746 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:40.565697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-btndx" event={"ID":"20bfb057-f2e5-4ee8-a0eb-ec32f856c558","Type":"ContainerStarted","Data":"5bc85e4a441580296a04f051f6bac76af9215a725e717a46c036f562b3adff30"}
Jul 08 23:16:40.565858 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:40.565827    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:40.593254 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:40.593214    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-btndx" podStartSLOduration=2.114935233 podStartE2EDuration="5.593200593s" totalImagesPullingTime="3.47826536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:16:40.591143913 +0000 UTC m=+549.582977220" watchObservedRunningTime="2026-07-08 23:16:40.593200593 +0000 UTC m=+549.585033932"
Jul 08 23:16:42.156572 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.156541    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"]
Jul 08 23:16:42.159673 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.159654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.164746 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.164723    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6tvpb\""
Jul 08 23:16:42.164746 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.164741    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 08 23:16:42.164922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.164727    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 08 23:16:42.164922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.164775    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 08 23:16:42.164922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.164728    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 08 23:16:42.168370 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.168355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 08 23:16:42.195301 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.195278    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"]
Jul 08 23:16:42.350313 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.350234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3964231-a862-4bdf-af43-029ad615875d-metrics-cert\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.350313 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.350309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3964231-a862-4bdf-af43-029ad615875d-cert\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.350565 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.350344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3964231-a862-4bdf-af43-029ad615875d-manager-config\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.350565 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.350437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nz8\" (UniqueName: \"kubernetes.io/projected/b3964231-a862-4bdf-af43-029ad615875d-kube-api-access-t9nz8\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.451894 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.451808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3964231-a862-4bdf-af43-029ad615875d-cert\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.451894 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.451869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3964231-a862-4bdf-af43-029ad615875d-manager-config\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.452083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.451942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nz8\" (UniqueName: \"kubernetes.io/projected/b3964231-a862-4bdf-af43-029ad615875d-kube-api-access-t9nz8\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.452083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.451986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3964231-a862-4bdf-af43-029ad615875d-metrics-cert\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.452597 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.452551    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3964231-a862-4bdf-af43-029ad615875d-manager-config\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.454668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.454644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3964231-a862-4bdf-af43-029ad615875d-cert\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.454849 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.454827    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3964231-a862-4bdf-af43-029ad615875d-metrics-cert\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.469254 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.469226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nz8\" (UniqueName: \"kubernetes.io/projected/b3964231-a862-4bdf-af43-029ad615875d-kube-api-access-t9nz8\") pod \"lws-controller-manager-56bdb7f565-9c2bw\" (UID: \"b3964231-a862-4bdf-af43-029ad615875d\") " pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:42.469381 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:42.469367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:43.085068 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:43.085045    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"]
Jul 08 23:16:43.089512 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:16:43.089485    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3964231_a862_4bdf_af43_029ad615875d.slice/crio-8548907ef5fc90efe47e74d2819460e74e7b49ce7b8f0dce682b345d8b8d25e0 WatchSource:0}: Error finding container 8548907ef5fc90efe47e74d2819460e74e7b49ce7b8f0dce682b345d8b8d25e0: Status 404 returned error can't find the container with id 8548907ef5fc90efe47e74d2819460e74e7b49ce7b8f0dce682b345d8b8d25e0
Jul 08 23:16:43.583481 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:43.582129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw" event={"ID":"b3964231-a862-4bdf-af43-029ad615875d","Type":"ContainerStarted","Data":"8548907ef5fc90efe47e74d2819460e74e7b49ce7b8f0dce682b345d8b8d25e0"}
Jul 08 23:16:43.584382 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:43.584350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259" event={"ID":"1ca08daf-44f5-406d-aa54-dc36507f0b47","Type":"ContainerStarted","Data":"2146db529c78613abe0969867ca971800cde66a5d5a84504ecc247a2feec1df1"}
Jul 08 23:16:45.593930 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:45.593855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw" event={"ID":"b3964231-a862-4bdf-af43-029ad615875d","Type":"ContainerStarted","Data":"e30f5d289136fae0267f6224ad8a0173551f16f73d055b55cec34565b69919a0"}
Jul 08 23:16:45.594256 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:45.594083    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:16:45.626874 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:45.626830    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b8998bd98-k9259" podStartSLOduration=4.946019243 podStartE2EDuration="7.626815527s" totalImagesPullingTime="2.680796284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:16:43.622433468 +0000 UTC m=+552.614266775" watchObservedRunningTime="2026-07-08 23:16:45.626815527 +0000 UTC m=+554.618648908"
Jul 08 23:16:45.627014 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:45.626903    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw" podStartSLOduration=1.493001285 podStartE2EDuration="3.626900297s" totalImagesPullingTime="2.133899012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:16:45.625374211 +0000 UTC m=+554.617207518" watchObservedRunningTime="2026-07-08 23:16:45.626900297 +0000 UTC m=+554.618733665"
Jul 08 23:16:51.571328 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:51.571303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-btndx"
Jul 08 23:16:56.599553 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:16:56.599519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56bdb7f565-9c2bw"
Jul 08 23:19:16.274393 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.274355    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qtcdm"]
Jul 08 23:19:16.277669 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.277648    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:16.280273 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.280250    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 23:19:16.281322 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.281302    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pswlt\""
Jul 08 23:19:16.281385 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.281307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 23:19:16.286412 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.286385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtcdm"]
Jul 08 23:19:16.341905 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.341879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcx8d\" (UniqueName: \"kubernetes.io/projected/89609058-73c9-49eb-ad8a-44e2f9a1b454-kube-api-access-dcx8d\") pod \"authorino-58cd955675-qtcdm\" (UID: \"89609058-73c9-49eb-ad8a-44e2f9a1b454\") " pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:16.442298 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.442270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcx8d\" (UniqueName: \"kubernetes.io/projected/89609058-73c9-49eb-ad8a-44e2f9a1b454-kube-api-access-dcx8d\") pod \"authorino-58cd955675-qtcdm\" (UID: \"89609058-73c9-49eb-ad8a-44e2f9a1b454\") " pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:16.450944 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.450922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcx8d\" (UniqueName: \"kubernetes.io/projected/89609058-73c9-49eb-ad8a-44e2f9a1b454-kube-api-access-dcx8d\") pod \"authorino-58cd955675-qtcdm\" (UID: \"89609058-73c9-49eb-ad8a-44e2f9a1b454\") " pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:16.468437 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.468410    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-sjm4r"]
Jul 08 23:19:16.471733 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.471687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:16.480738 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.480715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sjm4r"]
Jul 08 23:19:16.543502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.543449    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf5d9\" (UniqueName: \"kubernetes.io/projected/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893-kube-api-access-xf5d9\") pod \"authorino-75d556bd77-sjm4r\" (UID: \"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893\") " pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:16.588451 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.588423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:16.644518 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.644485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf5d9\" (UniqueName: \"kubernetes.io/projected/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893-kube-api-access-xf5d9\") pod \"authorino-75d556bd77-sjm4r\" (UID: \"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893\") " pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:16.653234 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.653187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf5d9\" (UniqueName: \"kubernetes.io/projected/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893-kube-api-access-xf5d9\") pod \"authorino-75d556bd77-sjm4r\" (UID: \"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893\") " pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:16.715175 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.715143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtcdm"]
Jul 08 23:19:16.718052 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:19:16.718023    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89609058_73c9_49eb_ad8a_44e2f9a1b454.slice/crio-2b85395e74ae83686e3b994e227be2003aab036b4ad2aa4b1ab8e8b9f78e7557 WatchSource:0}: Error finding container 2b85395e74ae83686e3b994e227be2003aab036b4ad2aa4b1ab8e8b9f78e7557: Status 404 returned error can't find the container with id 2b85395e74ae83686e3b994e227be2003aab036b4ad2aa4b1ab8e8b9f78e7557
Jul 08 23:19:16.719298 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.719282    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 23:19:16.780940 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.780911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:16.902249 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:16.902137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sjm4r"]
Jul 08 23:19:16.904648 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:19:16.904628    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0aae5c0_ab21_4f2a_a7ca_74fd47ec2893.slice/crio-a307ab5ad8f6371e6c14331f1c081cadec995c2f9b128a30b4f9f49f756b4d9d WatchSource:0}: Error finding container a307ab5ad8f6371e6c14331f1c081cadec995c2f9b128a30b4f9f49f756b4d9d: Status 404 returned error can't find the container with id a307ab5ad8f6371e6c14331f1c081cadec995c2f9b128a30b4f9f49f756b4d9d
Jul 08 23:19:17.231919 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:17.231889    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sjm4r" event={"ID":"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893","Type":"ContainerStarted","Data":"a307ab5ad8f6371e6c14331f1c081cadec995c2f9b128a30b4f9f49f756b4d9d"}
Jul 08 23:19:17.234874 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:17.234838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtcdm" event={"ID":"89609058-73c9-49eb-ad8a-44e2f9a1b454","Type":"ContainerStarted","Data":"2b85395e74ae83686e3b994e227be2003aab036b4ad2aa4b1ab8e8b9f78e7557"}
Jul 08 23:19:21.255960 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:21.255916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtcdm" event={"ID":"89609058-73c9-49eb-ad8a-44e2f9a1b454","Type":"ContainerStarted","Data":"854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a"}
Jul 08 23:19:21.257524 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:21.257497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sjm4r" event={"ID":"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893","Type":"ContainerStarted","Data":"ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c"}
Jul 08 23:19:21.270152 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:21.270112    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qtcdm" podStartSLOduration=1.761320402 podStartE2EDuration="5.270100458s" totalImagesPullingTime="3.508780056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:19:21.268617394 +0000 UTC m=+710.260450679" watchObservedRunningTime="2026-07-08 23:19:21.270100458 +0000 UTC m=+710.261933764"
Jul 08 23:19:21.282086 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:21.282042    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-sjm4r" podStartSLOduration=1.96973357 podStartE2EDuration="5.282029973s" totalImagesPullingTime="3.312296403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:19:21.281991663 +0000 UTC m=+710.273824970" watchObservedRunningTime="2026-07-08 23:19:21.282029973 +0000 UTC m=+710.273863277"
Jul 08 23:19:21.302847 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:21.302824    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtcdm"]
Jul 08 23:19:23.265867 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:23.265786    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qtcdm" podUID="89609058-73c9-49eb-ad8a-44e2f9a1b454" containerName="authorino" containerID="cri-o://854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a" gracePeriod=30
Jul 08 23:19:23.503064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:23.503041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:23.596841 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:23.596775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcx8d\" (UniqueName: \"kubernetes.io/projected/89609058-73c9-49eb-ad8a-44e2f9a1b454-kube-api-access-dcx8d\") pod \"89609058-73c9-49eb-ad8a-44e2f9a1b454\" (UID: \"89609058-73c9-49eb-ad8a-44e2f9a1b454\") "
Jul 08 23:19:23.598731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:23.598692    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89609058-73c9-49eb-ad8a-44e2f9a1b454-kube-api-access-dcx8d" (OuterVolumeSpecName: "kube-api-access-dcx8d") pod "89609058-73c9-49eb-ad8a-44e2f9a1b454" (UID: "89609058-73c9-49eb-ad8a-44e2f9a1b454"). InnerVolumeSpecName "kube-api-access-dcx8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:19:23.697828 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:23.697802    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcx8d\" (UniqueName: \"kubernetes.io/projected/89609058-73c9-49eb-ad8a-44e2f9a1b454-kube-api-access-dcx8d\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:19:24.270886 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.270853    2577 generic.go:356] "Generic (PLEG): container finished" podID="89609058-73c9-49eb-ad8a-44e2f9a1b454" containerID="854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a" exitCode=0
Jul 08 23:19:24.271302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.270896    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtcdm"
Jul 08 23:19:24.271302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.270944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtcdm" event={"ID":"89609058-73c9-49eb-ad8a-44e2f9a1b454","Type":"ContainerDied","Data":"854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a"}
Jul 08 23:19:24.271302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.270993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtcdm" event={"ID":"89609058-73c9-49eb-ad8a-44e2f9a1b454","Type":"ContainerDied","Data":"2b85395e74ae83686e3b994e227be2003aab036b4ad2aa4b1ab8e8b9f78e7557"}
Jul 08 23:19:24.271302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.271022    2577 scope.go:117] "RemoveContainer" containerID="854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a"
Jul 08 23:19:24.279012 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.278995    2577 scope.go:117] "RemoveContainer" containerID="854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a"
Jul 08 23:19:24.279285 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:19:24.279253    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a\": container with ID starting with 854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a not found: ID does not exist" containerID="854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a"
Jul 08 23:19:24.279324 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.279300    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a"} err="failed to get container status \"854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a\": rpc error: code = NotFound desc = could not find container \"854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a\": container with ID starting with 854c3efe5d14d894402452c56241f2bdee956731e54cdbdec69413517041c81a not found: ID does not exist"
Jul 08 23:19:24.290502 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.290483    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtcdm"]
Jul 08 23:19:24.294797 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:24.294776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtcdm"]
Jul 08 23:19:25.519874 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:25.519831    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89609058-73c9-49eb-ad8a-44e2f9a1b454" path="/var/lib/kubelet/pods/89609058-73c9-49eb-ad8a-44e2f9a1b454/volumes"
Jul 08 23:19:49.607082 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.607040    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kpzwq"]
Jul 08 23:19:49.607538 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.607328    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89609058-73c9-49eb-ad8a-44e2f9a1b454" containerName="authorino"
Jul 08 23:19:49.607538 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.607339    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89609058-73c9-49eb-ad8a-44e2f9a1b454" containerName="authorino"
Jul 08 23:19:49.607538 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.607393    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89609058-73c9-49eb-ad8a-44e2f9a1b454" containerName="authorino"
Jul 08 23:19:49.611416 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.611400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:49.616009 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.615990    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kpzwq"]
Jul 08 23:19:49.780954 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.780929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6q4\" (UniqueName: \"kubernetes.io/projected/d3b61c27-574f-4c91-88f0-c9e50a60ccd4-kube-api-access-6q6q4\") pod \"authorino-5d4b67c446-kpzwq\" (UID: \"d3b61c27-574f-4c91-88f0-c9e50a60ccd4\") " pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:49.847464 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.847432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kpzwq"]
Jul 08 23:19:49.847656 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:19:49.847634    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6q6q4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-kpzwq" podUID="d3b61c27-574f-4c91-88f0-c9e50a60ccd4"
Jul 08 23:19:49.872311 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.872285    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-789774d85d-khtz4"]
Jul 08 23:19:49.875253 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.875236    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:49.881279 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.881254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6q4\" (UniqueName: \"kubernetes.io/projected/d3b61c27-574f-4c91-88f0-c9e50a60ccd4-kube-api-access-6q6q4\") pod \"authorino-5d4b67c446-kpzwq\" (UID: \"d3b61c27-574f-4c91-88f0-c9e50a60ccd4\") " pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:49.882534 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.882515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-789774d85d-khtz4"]
Jul 08 23:19:49.888358 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.888334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6q4\" (UniqueName: \"kubernetes.io/projected/d3b61c27-574f-4c91-88f0-c9e50a60ccd4-kube-api-access-6q6q4\") pod \"authorino-5d4b67c446-kpzwq\" (UID: \"d3b61c27-574f-4c91-88f0-c9e50a60ccd4\") " pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:49.900111 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.900086    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-789774d85d-khtz4"]
Jul 08 23:19:49.900261 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:19:49.900244    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gb9d7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-789774d85d-khtz4" podUID="0f8af666-d73c-4636-a25f-450957cc672a"
Jul 08 23:19:49.927388 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.927362    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f47d45dcd-tsrhw"]
Jul 08 23:19:49.930393 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.930379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:49.932519 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.932499    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 08 23:19:49.941031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.941011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f47d45dcd-tsrhw"]
Jul 08 23:19:49.981880 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:49.981857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9d7\" (UniqueName: \"kubernetes.io/projected/0f8af666-d73c-4636-a25f-450957cc672a-kube-api-access-gb9d7\") pod \"authorino-789774d85d-khtz4\" (UID: \"0f8af666-d73c-4636-a25f-450957cc672a\") " pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:50.082791 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.082770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fpn\" (UniqueName: \"kubernetes.io/projected/52bda0ab-1151-4ce8-822b-9e1163076368-kube-api-access-r2fpn\") pod \"authorino-f47d45dcd-tsrhw\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") " pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.082891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.082810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9d7\" (UniqueName: \"kubernetes.io/projected/0f8af666-d73c-4636-a25f-450957cc672a-kube-api-access-gb9d7\") pod \"authorino-789774d85d-khtz4\" (UID: \"0f8af666-d73c-4636-a25f-450957cc672a\") " pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:50.082891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.082881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52bda0ab-1151-4ce8-822b-9e1163076368-tls-cert\") pod \"authorino-f47d45dcd-tsrhw\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") " pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.090761 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.090741    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9d7\" (UniqueName: \"kubernetes.io/projected/0f8af666-d73c-4636-a25f-450957cc672a-kube-api-access-gb9d7\") pod \"authorino-789774d85d-khtz4\" (UID: \"0f8af666-d73c-4636-a25f-450957cc672a\") " pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:50.184028 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.183975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fpn\" (UniqueName: \"kubernetes.io/projected/52bda0ab-1151-4ce8-822b-9e1163076368-kube-api-access-r2fpn\") pod \"authorino-f47d45dcd-tsrhw\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") " pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.184109 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.184032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52bda0ab-1151-4ce8-822b-9e1163076368-tls-cert\") pod \"authorino-f47d45dcd-tsrhw\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") " pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.186191 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.186171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52bda0ab-1151-4ce8-822b-9e1163076368-tls-cert\") pod \"authorino-f47d45dcd-tsrhw\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") " pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.191436 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.191417    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fpn\" (UniqueName: \"kubernetes.io/projected/52bda0ab-1151-4ce8-822b-9e1163076368-kube-api-access-r2fpn\") pod \"authorino-f47d45dcd-tsrhw\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") " pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.239343 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.239313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:19:50.359594 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.359570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f47d45dcd-tsrhw"]
Jul 08 23:19:50.361305 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:19:50.361278    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52bda0ab_1151_4ce8_822b_9e1163076368.slice/crio-3fd243ff950a9aadcf6aa1c424af7cf480d9acf619c39a39848f554a0e5c20d5 WatchSource:0}: Error finding container 3fd243ff950a9aadcf6aa1c424af7cf480d9acf619c39a39848f554a0e5c20d5: Status 404 returned error can't find the container with id 3fd243ff950a9aadcf6aa1c424af7cf480d9acf619c39a39848f554a0e5c20d5
Jul 08 23:19:50.383291 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.383258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f47d45dcd-tsrhw" event={"ID":"52bda0ab-1151-4ce8-822b-9e1163076368","Type":"ContainerStarted","Data":"3fd243ff950a9aadcf6aa1c424af7cf480d9acf619c39a39848f554a0e5c20d5"}
Jul 08 23:19:50.383383 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.383292    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:50.383383 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.383271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:50.391148 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.391134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:50.393235 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.393218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:50.486497 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.486454    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q6q4\" (UniqueName: \"kubernetes.io/projected/d3b61c27-574f-4c91-88f0-c9e50a60ccd4-kube-api-access-6q6q4\") pod \"d3b61c27-574f-4c91-88f0-c9e50a60ccd4\" (UID: \"d3b61c27-574f-4c91-88f0-c9e50a60ccd4\") "
Jul 08 23:19:50.486497 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.486488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb9d7\" (UniqueName: \"kubernetes.io/projected/0f8af666-d73c-4636-a25f-450957cc672a-kube-api-access-gb9d7\") pod \"0f8af666-d73c-4636-a25f-450957cc672a\" (UID: \"0f8af666-d73c-4636-a25f-450957cc672a\") "
Jul 08 23:19:50.488445 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.488425    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8af666-d73c-4636-a25f-450957cc672a-kube-api-access-gb9d7" (OuterVolumeSpecName: "kube-api-access-gb9d7") pod "0f8af666-d73c-4636-a25f-450957cc672a" (UID: "0f8af666-d73c-4636-a25f-450957cc672a"). InnerVolumeSpecName "kube-api-access-gb9d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:19:50.488445 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.488440    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b61c27-574f-4c91-88f0-c9e50a60ccd4-kube-api-access-6q6q4" (OuterVolumeSpecName: "kube-api-access-6q6q4") pod "d3b61c27-574f-4c91-88f0-c9e50a60ccd4" (UID: "d3b61c27-574f-4c91-88f0-c9e50a60ccd4"). InnerVolumeSpecName "kube-api-access-6q6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:19:50.587755 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.587733    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6q4\" (UniqueName: \"kubernetes.io/projected/d3b61c27-574f-4c91-88f0-c9e50a60ccd4-kube-api-access-6q6q4\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:19:50.587755 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:50.587752    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9d7\" (UniqueName: \"kubernetes.io/projected/0f8af666-d73c-4636-a25f-450957cc672a-kube-api-access-gb9d7\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:19:51.393899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.393871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-kpzwq"
Jul 08 23:19:51.394283 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.393912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f47d45dcd-tsrhw" event={"ID":"52bda0ab-1151-4ce8-822b-9e1163076368","Type":"ContainerStarted","Data":"573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e"}
Jul 08 23:19:51.394283 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.394128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-789774d85d-khtz4"
Jul 08 23:19:51.410443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.410399    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f47d45dcd-tsrhw" podStartSLOduration=2.014006945 podStartE2EDuration="2.410386717s" totalImagesPullingTime="396.379772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:19:51.409854385 +0000 UTC m=+740.401687692" watchObservedRunningTime="2026-07-08 23:19:51.410386717 +0000 UTC m=+740.402220062"
Jul 08 23:19:51.432896 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.432873    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kpzwq"]
Jul 08 23:19:51.438578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.438560    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sjm4r"]
Jul 08 23:19:51.438731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.438695    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-sjm4r" podUID="e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" containerName="authorino" containerID="cri-o://ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c" gracePeriod=30
Jul 08 23:19:51.441463 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.441430    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-kpzwq"]
Jul 08 23:19:51.461885 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.461865    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-789774d85d-khtz4"]
Jul 08 23:19:51.463789 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.463771    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-789774d85d-khtz4"]
Jul 08 23:19:51.519785 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.519765    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8af666-d73c-4636-a25f-450957cc672a" path="/var/lib/kubelet/pods/0f8af666-d73c-4636-a25f-450957cc672a/volumes"
Jul 08 23:19:51.519997 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.519985    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3b61c27-574f-4c91-88f0-c9e50a60ccd4" path="/var/lib/kubelet/pods/d3b61c27-574f-4c91-88f0-c9e50a60ccd4/volumes"
Jul 08 23:19:51.677863 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.677841    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:51.696260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.696236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf5d9\" (UniqueName: \"kubernetes.io/projected/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893-kube-api-access-xf5d9\") pod \"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893\" (UID: \"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893\") "
Jul 08 23:19:51.698257 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.698229    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893-kube-api-access-xf5d9" (OuterVolumeSpecName: "kube-api-access-xf5d9") pod "e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" (UID: "e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893"). InnerVolumeSpecName "kube-api-access-xf5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:19:51.796973 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:51.796948    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf5d9\" (UniqueName: \"kubernetes.io/projected/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893-kube-api-access-xf5d9\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:19:52.400488 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.400454    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" containerID="ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c" exitCode=0
Jul 08 23:19:52.400899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.400503    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sjm4r"
Jul 08 23:19:52.400899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.400539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sjm4r" event={"ID":"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893","Type":"ContainerDied","Data":"ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c"}
Jul 08 23:19:52.400899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.400591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sjm4r" event={"ID":"e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893","Type":"ContainerDied","Data":"a307ab5ad8f6371e6c14331f1c081cadec995c2f9b128a30b4f9f49f756b4d9d"}
Jul 08 23:19:52.400899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.400609    2577 scope.go:117] "RemoveContainer" containerID="ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c"
Jul 08 23:19:52.410420 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.410404    2577 scope.go:117] "RemoveContainer" containerID="ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c"
Jul 08 23:19:52.410715 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:19:52.410681    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c\": container with ID starting with ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c not found: ID does not exist" containerID="ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c"
Jul 08 23:19:52.410766 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.410744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c"} err="failed to get container status \"ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c\": rpc error: code = NotFound desc = could not find container \"ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c\": container with ID starting with ca63dc39d6adb1d26b0cbe03d9ee964e8dc3bbe2e272cd8f6a92c5c9e37c3f1c not found: ID does not exist"
Jul 08 23:19:52.420204 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.420176    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sjm4r"]
Jul 08 23:19:52.423599 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:52.423575    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sjm4r"]
Jul 08 23:19:53.518939 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:53.518911    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" path="/var/lib/kubelet/pods/e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893/volumes"
Jul 08 23:19:55.698022 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.697992    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5786d796df-jkkmn"]
Jul 08 23:19:55.698484 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.698442    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" containerName="authorino"
Jul 08 23:19:55.698484 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.698460    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" containerName="authorino"
Jul 08 23:19:55.698607 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.698592    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0aae5c0-ab21-4f2a-a7ca-74fd47ec2893" containerName="authorino"
Jul 08 23:19:55.702747 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.702725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:55.705305 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.705283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 08 23:19:55.705423 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.705370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xs4mv\""
Jul 08 23:19:55.706444 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.706428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 08 23:19:55.710030 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.710013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5786d796df-jkkmn"]
Jul 08 23:19:55.721887 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.721866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2njpv\" (UniqueName: \"kubernetes.io/projected/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-kube-api-access-2njpv\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:55.721985 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.721924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:55.822371 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.822353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2njpv\" (UniqueName: \"kubernetes.io/projected/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-kube-api-access-2njpv\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:55.822459 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.822397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:55.822516 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:19:55.822481    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 08 23:19:55.822554 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:19:55.822526    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert podName:73c9eb7d-ac55-40e3-87aa-35a9b7668f7c nodeName:}" failed. No retries permitted until 2026-07-08 23:19:56.322511161 +0000 UTC m=+745.314344446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert") pod "maas-controller-5786d796df-jkkmn" (UID: "73c9eb7d-ac55-40e3-87aa-35a9b7668f7c") : secret "maas-controller-webhook-cert" not found
Jul 08 23:19:55.831261 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:55.831236    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2njpv\" (UniqueName: \"kubernetes.io/projected/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-kube-api-access-2njpv\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:56.326365 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:56.326334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:56.328894 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:56.328869    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert\") pod \"maas-controller-5786d796df-jkkmn\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") " pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:56.613511 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:56.613489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:56.752144 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:56.752116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5786d796df-jkkmn"]
Jul 08 23:19:56.754456 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:19:56.754432    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c9eb7d_ac55_40e3_87aa_35a9b7668f7c.slice/crio-5f2c87524237b8e1598039b118c75a51c9a4c3da7a3375396de52ec4b4c90f5b WatchSource:0}: Error finding container 5f2c87524237b8e1598039b118c75a51c9a4c3da7a3375396de52ec4b4c90f5b: Status 404 returned error can't find the container with id 5f2c87524237b8e1598039b118c75a51c9a4c3da7a3375396de52ec4b4c90f5b
Jul 08 23:19:57.426663 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:57.426624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5786d796df-jkkmn" event={"ID":"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c","Type":"ContainerStarted","Data":"5f2c87524237b8e1598039b118c75a51c9a4c3da7a3375396de52ec4b4c90f5b"}
Jul 08 23:19:59.436170 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:59.436132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5786d796df-jkkmn" event={"ID":"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c","Type":"ContainerStarted","Data":"7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654"}
Jul 08 23:19:59.436552 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:59.436249    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:19:59.452419 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:19:59.452379    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5786d796df-jkkmn" podStartSLOduration=2.7843999200000003 podStartE2EDuration="4.452366808s" totalImagesPullingTime="1.667966888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:19:59.451359253 +0000 UTC m=+748.443192560" watchObservedRunningTime="2026-07-08 23:19:59.452366808 +0000 UTC m=+748.444200114"
Jul 08 23:20:10.179096 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.179065    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5786d796df-jkkmn"]
Jul 08 23:20:10.179552 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.179246    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-5786d796df-jkkmn" podUID="73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" containerName="manager" containerID="cri-o://7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654" gracePeriod=30
Jul 08 23:20:10.185633 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.185609    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:20:10.411378 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.411357    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:20:10.426312 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.426291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert\") pod \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") "
Jul 08 23:20:10.426371 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.426345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2njpv\" (UniqueName: \"kubernetes.io/projected/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-kube-api-access-2njpv\") pod \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\" (UID: \"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c\") "
Jul 08 23:20:10.428890 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.428830    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" (UID: "73c9eb7d-ac55-40e3-87aa-35a9b7668f7c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:20:10.428999 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.428946    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-kube-api-access-2njpv" (OuterVolumeSpecName: "kube-api-access-2njpv") pod "73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" (UID: "73c9eb7d-ac55-40e3-87aa-35a9b7668f7c"). InnerVolumeSpecName "kube-api-access-2njpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:20:10.485656 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.485630    2577 generic.go:356] "Generic (PLEG): container finished" podID="73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" containerID="7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654" exitCode=0
Jul 08 23:20:10.485778 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.485673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5786d796df-jkkmn"
Jul 08 23:20:10.485778 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.485736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5786d796df-jkkmn" event={"ID":"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c","Type":"ContainerDied","Data":"7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654"}
Jul 08 23:20:10.485852 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.485813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5786d796df-jkkmn" event={"ID":"73c9eb7d-ac55-40e3-87aa-35a9b7668f7c","Type":"ContainerDied","Data":"5f2c87524237b8e1598039b118c75a51c9a4c3da7a3375396de52ec4b4c90f5b"}
Jul 08 23:20:10.485852 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.485839    2577 scope.go:117] "RemoveContainer" containerID="7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654"
Jul 08 23:20:10.494260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.494245    2577 scope.go:117] "RemoveContainer" containerID="7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654"
Jul 08 23:20:10.494513 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:20:10.494495    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654\": container with ID starting with 7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654 not found: ID does not exist" containerID="7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654"
Jul 08 23:20:10.494557 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.494521    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654"} err="failed to get container status \"7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654\": rpc error: code = NotFound desc = could not find container \"7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654\": container with ID starting with 7eb62eb8f9ad11fe3110a7526b0c20445df61a73f9c08abb6202f74d32eae654 not found: ID does not exist"
Jul 08 23:20:10.504904 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.504885    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5786d796df-jkkmn"]
Jul 08 23:20:10.509810 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.509786    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5786d796df-jkkmn"]
Jul 08 23:20:10.527472 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.527448    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-webhook-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:20:10.527472 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:10.527469    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2njpv\" (UniqueName: \"kubernetes.io/projected/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c-kube-api-access-2njpv\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:20:11.519122 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:11.519090    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" path="/var/lib/kubelet/pods/73c9eb7d-ac55-40e3-87aa-35a9b7668f7c/volumes"
Jul 08 23:20:30.489535 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.489504    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"]
Jul 08 23:20:30.490070 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.489804    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" containerName="manager"
Jul 08 23:20:30.490070 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.489816    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" containerName="manager"
Jul 08 23:20:30.490070 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.489869    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c9eb7d-ac55-40e3-87aa-35a9b7668f7c" containerName="manager"
Jul 08 23:20:30.492236 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.492219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.494776 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.494751    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bs8m\""
Jul 08 23:20:30.494881 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.494810    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 08 23:20:30.495765 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.495746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 08 23:20:30.495885 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.495778    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 08 23:20:30.503012 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.502992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"]
Jul 08 23:20:30.557835 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.557812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabbb9e-69a6-4168-a2e9-2a467806e501-maas-api-tls\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.557926 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.557840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabbb9e-69a6-4168-a2e9-2a467806e501-openshift-service-ca\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.557926 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.557871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6497w\" (UniqueName: \"kubernetes.io/projected/3fabbb9e-69a6-4168-a2e9-2a467806e501-kube-api-access-6497w\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.658436 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.658410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6497w\" (UniqueName: \"kubernetes.io/projected/3fabbb9e-69a6-4168-a2e9-2a467806e501-kube-api-access-6497w\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.658554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.658465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabbb9e-69a6-4168-a2e9-2a467806e501-maas-api-tls\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.658554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.658489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabbb9e-69a6-4168-a2e9-2a467806e501-openshift-service-ca\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.658663 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:20:30.658575    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 08 23:20:30.658751 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:20:30.658671    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fabbb9e-69a6-4168-a2e9-2a467806e501-maas-api-tls podName:3fabbb9e-69a6-4168-a2e9-2a467806e501 nodeName:}" failed. No retries permitted until 2026-07-08 23:20:31.1586495 +0000 UTC m=+780.150482793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fabbb9e-69a6-4168-a2e9-2a467806e501-maas-api-tls") pod "maas-api-5f6c8885dd-pjlcp" (UID: "3fabbb9e-69a6-4168-a2e9-2a467806e501") : secret "maas-api-serving-cert" not found
Jul 08 23:20:30.659136 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.659119    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fabbb9e-69a6-4168-a2e9-2a467806e501-openshift-service-ca\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.666472 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.666454    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6497w\" (UniqueName: \"kubernetes.io/projected/3fabbb9e-69a6-4168-a2e9-2a467806e501-kube-api-access-6497w\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:30.978715 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.978672    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-8t7tb"]
Jul 08 23:20:30.980800 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.980784    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:30.983178 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.983158    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fxj27\""
Jul 08 23:20:30.983520 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.983502    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 08 23:20:30.999733 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:30.999693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-8t7tb"]
Jul 08 23:20:31.062143 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.062115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst6w\" (UniqueName: \"kubernetes.io/projected/adef2d3c-bde5-4788-8fa8-dc967afb75ea-kube-api-access-dst6w\") pod \"payload-pre-processing-7964457696-8t7tb\" (UID: \"adef2d3c-bde5-4788-8fa8-dc967afb75ea\") " pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.062252 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.062231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef2d3c-bde5-4788-8fa8-dc967afb75ea-plugins-config-volume\") pod \"payload-pre-processing-7964457696-8t7tb\" (UID: \"adef2d3c-bde5-4788-8fa8-dc967afb75ea\") " pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.163364 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.163330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabbb9e-69a6-4168-a2e9-2a467806e501-maas-api-tls\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:31.163470 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.163370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef2d3c-bde5-4788-8fa8-dc967afb75ea-plugins-config-volume\") pod \"payload-pre-processing-7964457696-8t7tb\" (UID: \"adef2d3c-bde5-4788-8fa8-dc967afb75ea\") " pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.163470 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.163428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst6w\" (UniqueName: \"kubernetes.io/projected/adef2d3c-bde5-4788-8fa8-dc967afb75ea-kube-api-access-dst6w\") pod \"payload-pre-processing-7964457696-8t7tb\" (UID: \"adef2d3c-bde5-4788-8fa8-dc967afb75ea\") " pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.164203 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.164180    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adef2d3c-bde5-4788-8fa8-dc967afb75ea-plugins-config-volume\") pod \"payload-pre-processing-7964457696-8t7tb\" (UID: \"adef2d3c-bde5-4788-8fa8-dc967afb75ea\") " pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.165865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.165844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fabbb9e-69a6-4168-a2e9-2a467806e501-maas-api-tls\") pod \"maas-api-5f6c8885dd-pjlcp\" (UID: \"3fabbb9e-69a6-4168-a2e9-2a467806e501\") " pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:31.170648 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.170630    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst6w\" (UniqueName: \"kubernetes.io/projected/adef2d3c-bde5-4788-8fa8-dc967afb75ea-kube-api-access-dst6w\") pod \"payload-pre-processing-7964457696-8t7tb\" (UID: \"adef2d3c-bde5-4788-8fa8-dc967afb75ea\") " pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.290736 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.290664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:31.404289 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.404263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9bs8m\""
Jul 08 23:20:31.410543 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.410526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-8t7tb"]
Jul 08 23:20:31.412269 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.412252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:31.412786 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:20:31.412762    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadef2d3c_bde5_4788_8fa8_dc967afb75ea.slice/crio-d72edc3c15c3e2f3ba7ca7c1c6ba3e663fc43c3c88a7990a6d8a51f1da2ecf89 WatchSource:0}: Error finding container d72edc3c15c3e2f3ba7ca7c1c6ba3e663fc43c3c88a7990a6d8a51f1da2ecf89: Status 404 returned error can't find the container with id d72edc3c15c3e2f3ba7ca7c1c6ba3e663fc43c3c88a7990a6d8a51f1da2ecf89
Jul 08 23:20:31.535296 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.535261    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"]
Jul 08 23:20:31.537592 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:20:31.537561    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fabbb9e_69a6_4168_a2e9_2a467806e501.slice/crio-81905c1353d514085e51598d9ea87f7e2f42403b6667b2120d508561a7bf8636 WatchSource:0}: Error finding container 81905c1353d514085e51598d9ea87f7e2f42403b6667b2120d508561a7bf8636: Status 404 returned error can't find the container with id 81905c1353d514085e51598d9ea87f7e2f42403b6667b2120d508561a7bf8636
Jul 08 23:20:31.575768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.575671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb" event={"ID":"adef2d3c-bde5-4788-8fa8-dc967afb75ea","Type":"ContainerStarted","Data":"d72edc3c15c3e2f3ba7ca7c1c6ba3e663fc43c3c88a7990a6d8a51f1da2ecf89"}
Jul 08 23:20:31.576681 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:31.576663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp" event={"ID":"3fabbb9e-69a6-4168-a2e9-2a467806e501","Type":"ContainerStarted","Data":"81905c1353d514085e51598d9ea87f7e2f42403b6667b2120d508561a7bf8636"}
Jul 08 23:20:33.586578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:33.586489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp" event={"ID":"3fabbb9e-69a6-4168-a2e9-2a467806e501","Type":"ContainerStarted","Data":"8bb17551f8dd78b8b52a271075e3f343a167f02d8c70a25398471934e353c126"}
Jul 08 23:20:33.586975 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:33.586654    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:33.587978 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:33.587954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb" event={"ID":"adef2d3c-bde5-4788-8fa8-dc967afb75ea","Type":"ContainerStarted","Data":"bb45cb6eb5bbd68c04fccc9b3e120567818ad81083f1356092d07f8c0cc0b32c"}
Jul 08 23:20:33.588101 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:33.588003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:20:33.608977 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:33.608938    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp" podStartSLOduration=1.8185483919999998 podStartE2EDuration="3.60892468s" totalImagesPullingTime="1.790376288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:20:33.606661968 +0000 UTC m=+782.598495277" watchObservedRunningTime="2026-07-08 23:20:33.60892468 +0000 UTC m=+782.600757985"
Jul 08 23:20:33.630536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:33.630500    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb" podStartSLOduration=2.205597109 podStartE2EDuration="3.630486425s" totalImagesPullingTime="1.424889316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:20:33.629024566 +0000 UTC m=+782.620857874" watchObservedRunningTime="2026-07-08 23:20:33.630486425 +0000 UTC m=+782.622319731"
Jul 08 23:20:39.598983 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:39.598949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5f6c8885dd-pjlcp"
Jul 08 23:20:44.594442 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:20:44.594413    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-8t7tb"
Jul 08 23:21:14.405312 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.405273    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"]
Jul 08 23:21:14.409995 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.409970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.412848 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.412821    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 08 23:21:14.413884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.413857    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nzknt\""
Jul 08 23:21:14.414002 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.413922    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 08 23:21:14.414002 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.413992    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 08 23:21:14.421332 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.421297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"]
Jul 08 23:21:14.584262 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.584237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3c2277-789b-45b6-bdbd-0a2f28f2170e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.584409 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.584281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.584409 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.584345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.584409 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.584397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6k5\" (UniqueName: \"kubernetes.io/projected/be3c2277-789b-45b6-bdbd-0a2f28f2170e-kube-api-access-7c6k5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.584554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.584431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.584554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.584458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685100 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685100 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685075    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685272 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6k5\" (UniqueName: \"kubernetes.io/projected/be3c2277-789b-45b6-bdbd-0a2f28f2170e-kube-api-access-7c6k5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685272 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685272 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685272 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3c2277-789b-45b6-bdbd-0a2f28f2170e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685466 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685443    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685546 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685524    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.685546 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.685538    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.687364 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.687343    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be3c2277-789b-45b6-bdbd-0a2f28f2170e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.687587 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.687573    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be3c2277-789b-45b6-bdbd-0a2f28f2170e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.692655 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.692633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6k5\" (UniqueName: \"kubernetes.io/projected/be3c2277-789b-45b6-bdbd-0a2f28f2170e-kube-api-access-7c6k5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tgfd4\" (UID: \"be3c2277-789b-45b6-bdbd-0a2f28f2170e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.721209 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.721188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:14.850838 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:14.850810    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"]
Jul 08 23:21:14.853695 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:21:14.853665    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3c2277_789b_45b6_bdbd_0a2f28f2170e.slice/crio-63534c1c43de54c562ee9ba5776bca4ebc805ccbd4e7065947219daa044d17cc WatchSource:0}: Error finding container 63534c1c43de54c562ee9ba5776bca4ebc805ccbd4e7065947219daa044d17cc: Status 404 returned error can't find the container with id 63534c1c43de54c562ee9ba5776bca4ebc805ccbd4e7065947219daa044d17cc
Jul 08 23:21:15.770516 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:15.770470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4" event={"ID":"be3c2277-789b-45b6-bdbd-0a2f28f2170e","Type":"ContainerStarted","Data":"63534c1c43de54c562ee9ba5776bca4ebc805ccbd4e7065947219daa044d17cc"}
Jul 08 23:21:16.776605 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:16.776565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4" event={"ID":"be3c2277-789b-45b6-bdbd-0a2f28f2170e","Type":"ContainerStarted","Data":"0310e2795ab579454d2fd6efe797dc6b988aa369bb7b633c25a912d47fcb26e8"}
Jul 08 23:21:16.796324 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:16.796275    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4" podStartSLOduration=1.775768302 podStartE2EDuration="2.796261202s" totalImagesPullingTime="1.0204929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:21:16.794429613 +0000 UTC m=+825.786262923" watchObservedRunningTime="2026-07-08 23:21:16.796261202 +0000 UTC m=+825.788094507"
Jul 08 23:21:24.722049 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:24.722019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:24.722515 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:24.722114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:24.734539 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:24.734518    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:24.825236 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:24.825210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tgfd4"
Jul 08 23:21:26.901357 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:26.901325    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"]
Jul 08 23:21:26.904291 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:26.904274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:26.906551 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:26.906524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 08 23:21:26.912688 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:26.912666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"]
Jul 08 23:21:27.074550 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.074524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e4b150-a17f-41de-b111-adf33fa6ff15-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.074692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.074583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.074692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.074610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.074692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.074646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpjr\" (UniqueName: \"kubernetes.io/projected/d5e4b150-a17f-41de-b111-adf33fa6ff15-kube-api-access-dxpjr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.074692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.074675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.074912 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.074782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.175902 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.175837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.175902 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.175876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e4b150-a17f-41de-b111-adf33fa6ff15-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176056 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.175913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176056 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.175932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176056 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.175964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpjr\" (UniqueName: \"kubernetes.io/projected/d5e4b150-a17f-41de-b111-adf33fa6ff15-kube-api-access-dxpjr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176056 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.175982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176321 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.176298    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176391 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.176349    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.176450 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.176395    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.178247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.178222    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e4b150-a17f-41de-b111-adf33fa6ff15-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.178392 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.178376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e4b150-a17f-41de-b111-adf33fa6ff15-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.185479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.185455    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpjr\" (UniqueName: \"kubernetes.io/projected/d5e4b150-a17f-41de-b111-adf33fa6ff15-kube-api-access-dxpjr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx\" (UID: \"d5e4b150-a17f-41de-b111-adf33fa6ff15\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.216375 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.216352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:27.343420 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.343394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"]
Jul 08 23:21:27.345621 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:21:27.345596    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e4b150_a17f_41de_b111_adf33fa6ff15.slice/crio-4ecdea6b1a55ce61bbf99b22a3720bf591bb5e3a4afb327be15e5efd6651ef96 WatchSource:0}: Error finding container 4ecdea6b1a55ce61bbf99b22a3720bf591bb5e3a4afb327be15e5efd6651ef96: Status 404 returned error can't find the container with id 4ecdea6b1a55ce61bbf99b22a3720bf591bb5e3a4afb327be15e5efd6651ef96
Jul 08 23:21:27.828104 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.828034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx" event={"ID":"d5e4b150-a17f-41de-b111-adf33fa6ff15","Type":"ContainerStarted","Data":"a658f528c378481ec3a7a29fc3eecbc34f01fbee3aa40add00c3426ad644b0dc"}
Jul 08 23:21:27.828104 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.828094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx" event={"ID":"d5e4b150-a17f-41de-b111-adf33fa6ff15","Type":"ContainerStarted","Data":"4ecdea6b1a55ce61bbf99b22a3720bf591bb5e3a4afb327be15e5efd6651ef96"}
Jul 08 23:21:27.844927 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:27.844881    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx" podStartSLOduration=1.665469308 podStartE2EDuration="1.84486726s" totalImagesPullingTime="179.397952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:21:27.843590929 +0000 UTC m=+836.835424234" watchObservedRunningTime="2026-07-08 23:21:27.84486726 +0000 UTC m=+836.836700565"
Jul 08 23:21:37.216662 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:37.216625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:37.217302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:37.216698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:37.229628 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:37.229605    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:21:37.889975 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:21:37.889946    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx"
Jul 08 23:22:44.269918 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.269877    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-786479bc9f-rn482"]
Jul 08 23:22:44.273300 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.273277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.280779 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.280755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-786479bc9f-rn482"]
Jul 08 23:22:44.441228 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.441200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmc5\" (UniqueName: \"kubernetes.io/projected/bbcc4057-726f-4b4e-be16-d95d620d748a-kube-api-access-6gmc5\") pod \"authorino-786479bc9f-rn482\" (UID: \"bbcc4057-726f-4b4e-be16-d95d620d748a\") " pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.441330 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.441241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbcc4057-726f-4b4e-be16-d95d620d748a-tls-cert\") pod \"authorino-786479bc9f-rn482\" (UID: \"bbcc4057-726f-4b4e-be16-d95d620d748a\") " pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.542165 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.542090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmc5\" (UniqueName: \"kubernetes.io/projected/bbcc4057-726f-4b4e-be16-d95d620d748a-kube-api-access-6gmc5\") pod \"authorino-786479bc9f-rn482\" (UID: \"bbcc4057-726f-4b4e-be16-d95d620d748a\") " pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.542165 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.542145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbcc4057-726f-4b4e-be16-d95d620d748a-tls-cert\") pod \"authorino-786479bc9f-rn482\" (UID: \"bbcc4057-726f-4b4e-be16-d95d620d748a\") " pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.544808 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.544783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bbcc4057-726f-4b4e-be16-d95d620d748a-tls-cert\") pod \"authorino-786479bc9f-rn482\" (UID: \"bbcc4057-726f-4b4e-be16-d95d620d748a\") " pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.557106 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.557081    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmc5\" (UniqueName: \"kubernetes.io/projected/bbcc4057-726f-4b4e-be16-d95d620d748a-kube-api-access-6gmc5\") pod \"authorino-786479bc9f-rn482\" (UID: \"bbcc4057-726f-4b4e-be16-d95d620d748a\") " pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.582743 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.582695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-786479bc9f-rn482"
Jul 08 23:22:44.717330 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:44.717302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-786479bc9f-rn482"]
Jul 08 23:22:44.719109 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:22:44.719080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbcc4057_726f_4b4e_be16_d95d620d748a.slice/crio-6e4083e961668d5b39fc1b1748ac9807ace41150df1378fec5c0b230ea381ec7 WatchSource:0}: Error finding container 6e4083e961668d5b39fc1b1748ac9807ace41150df1378fec5c0b230ea381ec7: Status 404 returned error can't find the container with id 6e4083e961668d5b39fc1b1748ac9807ace41150df1378fec5c0b230ea381ec7
Jul 08 23:22:45.176940 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:45.176903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-786479bc9f-rn482" event={"ID":"bbcc4057-726f-4b4e-be16-d95d620d748a","Type":"ContainerStarted","Data":"6e4083e961668d5b39fc1b1748ac9807ace41150df1378fec5c0b230ea381ec7"}
Jul 08 23:22:46.183852 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.183810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-786479bc9f-rn482" event={"ID":"bbcc4057-726f-4b4e-be16-d95d620d748a","Type":"ContainerStarted","Data":"52656ed17a80cc23a414964c64b8d55a99353bcdfd3770e4ef694823d40127a6"}
Jul 08 23:22:46.226389 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.226326    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-786479bc9f-rn482" podStartSLOduration=1.663593589 podStartE2EDuration="2.226302009s" totalImagesPullingTime="562.70842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:22:46.201988146 +0000 UTC m=+915.193821663" watchObservedRunningTime="2026-07-08 23:22:46.226302009 +0000 UTC m=+915.218135315"
Jul 08 23:22:46.227945 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.227917    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f47d45dcd-tsrhw"]
Jul 08 23:22:46.228295 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.228271    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-f47d45dcd-tsrhw" podUID="52bda0ab-1151-4ce8-822b-9e1163076368" containerName="authorino" containerID="cri-o://573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e" gracePeriod=30
Jul 08 23:22:46.476939 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.476907    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:22:46.663680 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.663650    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2fpn\" (UniqueName: \"kubernetes.io/projected/52bda0ab-1151-4ce8-822b-9e1163076368-kube-api-access-r2fpn\") pod \"52bda0ab-1151-4ce8-822b-9e1163076368\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") "
Jul 08 23:22:46.663884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.663761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52bda0ab-1151-4ce8-822b-9e1163076368-tls-cert\") pod \"52bda0ab-1151-4ce8-822b-9e1163076368\" (UID: \"52bda0ab-1151-4ce8-822b-9e1163076368\") "
Jul 08 23:22:46.665817 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.665787    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bda0ab-1151-4ce8-822b-9e1163076368-kube-api-access-r2fpn" (OuterVolumeSpecName: "kube-api-access-r2fpn") pod "52bda0ab-1151-4ce8-822b-9e1163076368" (UID: "52bda0ab-1151-4ce8-822b-9e1163076368"). InnerVolumeSpecName "kube-api-access-r2fpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:22:46.675590 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.675554    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52bda0ab-1151-4ce8-822b-9e1163076368-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "52bda0ab-1151-4ce8-822b-9e1163076368" (UID: "52bda0ab-1151-4ce8-822b-9e1163076368"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:22:46.764891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.764822    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/52bda0ab-1151-4ce8-822b-9e1163076368-tls-cert\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:22:46.764891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:46.764847    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2fpn\" (UniqueName: \"kubernetes.io/projected/52bda0ab-1151-4ce8-822b-9e1163076368-kube-api-access-r2fpn\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:22:47.189970 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.189932    2577 generic.go:356] "Generic (PLEG): container finished" podID="52bda0ab-1151-4ce8-822b-9e1163076368" containerID="573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e" exitCode=0
Jul 08 23:22:47.190368 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.189978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f47d45dcd-tsrhw"
Jul 08 23:22:47.190368 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.190016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f47d45dcd-tsrhw" event={"ID":"52bda0ab-1151-4ce8-822b-9e1163076368","Type":"ContainerDied","Data":"573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e"}
Jul 08 23:22:47.190368 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.190067    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f47d45dcd-tsrhw" event={"ID":"52bda0ab-1151-4ce8-822b-9e1163076368","Type":"ContainerDied","Data":"3fd243ff950a9aadcf6aa1c424af7cf480d9acf619c39a39848f554a0e5c20d5"}
Jul 08 23:22:47.190368 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.190086    2577 scope.go:117] "RemoveContainer" containerID="573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e"
Jul 08 23:22:47.199246 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.199227    2577 scope.go:117] "RemoveContainer" containerID="573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e"
Jul 08 23:22:47.199518 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:22:47.199500    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e\": container with ID starting with 573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e not found: ID does not exist" containerID="573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e"
Jul 08 23:22:47.199580 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.199524    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e"} err="failed to get container status \"573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e\": rpc error: code = NotFound desc = could not find container \"573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e\": container with ID starting with 573dea838373c854daad83000f4645f3cab53c0bbcaf61429c7901589b0b7f0e not found: ID does not exist"
Jul 08 23:22:47.210088 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.210058    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f47d45dcd-tsrhw"]
Jul 08 23:22:47.211829 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.211807    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f47d45dcd-tsrhw"]
Jul 08 23:22:47.520264 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:22:47.520191    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52bda0ab-1151-4ce8-822b-9e1163076368" path="/var/lib/kubelet/pods/52bda0ab-1151-4ce8-822b-9e1163076368/volumes"
Jul 08 23:23:58.043590 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.043554    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7464f4d4d-zzkmx"]
Jul 08 23:23:58.044182 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.044011    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bda0ab-1151-4ce8-822b-9e1163076368" containerName="authorino"
Jul 08 23:23:58.044182 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.044030    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bda0ab-1151-4ce8-822b-9e1163076368" containerName="authorino"
Jul 08 23:23:58.044182 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.044104    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bda0ab-1151-4ce8-822b-9e1163076368" containerName="authorino"
Jul 08 23:23:58.047105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.047086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.050928 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.050898    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 08 23:23:58.051108 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.050926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xs4mv\""
Jul 08 23:23:58.051108 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.050903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 08 23:23:58.057976 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.057951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7464f4d4d-zzkmx"]
Jul 08 23:23:58.200550 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.200522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4szk\" (UniqueName: \"kubernetes.io/projected/9b95ca82-e911-482a-ae68-d49d2c641dc1-kube-api-access-f4szk\") pod \"maas-controller-7464f4d4d-zzkmx\" (UID: \"9b95ca82-e911-482a-ae68-d49d2c641dc1\") " pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.200657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.200577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b95ca82-e911-482a-ae68-d49d2c641dc1-webhook-cert\") pod \"maas-controller-7464f4d4d-zzkmx\" (UID: \"9b95ca82-e911-482a-ae68-d49d2c641dc1\") " pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.301414 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.301362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b95ca82-e911-482a-ae68-d49d2c641dc1-webhook-cert\") pod \"maas-controller-7464f4d4d-zzkmx\" (UID: \"9b95ca82-e911-482a-ae68-d49d2c641dc1\") " pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.301414 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.301411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4szk\" (UniqueName: \"kubernetes.io/projected/9b95ca82-e911-482a-ae68-d49d2c641dc1-kube-api-access-f4szk\") pod \"maas-controller-7464f4d4d-zzkmx\" (UID: \"9b95ca82-e911-482a-ae68-d49d2c641dc1\") " pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.303804 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.303777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b95ca82-e911-482a-ae68-d49d2c641dc1-webhook-cert\") pod \"maas-controller-7464f4d4d-zzkmx\" (UID: \"9b95ca82-e911-482a-ae68-d49d2c641dc1\") " pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.309273 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.309250    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4szk\" (UniqueName: \"kubernetes.io/projected/9b95ca82-e911-482a-ae68-d49d2c641dc1-kube-api-access-f4szk\") pod \"maas-controller-7464f4d4d-zzkmx\" (UID: \"9b95ca82-e911-482a-ae68-d49d2c641dc1\") " pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.358339 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.358317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:58.483082 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.483056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7464f4d4d-zzkmx"]
Jul 08 23:23:58.485084 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:23:58.485057    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b95ca82_e911_482a_ae68_d49d2c641dc1.slice/crio-b72fd09554904f2217741930c1ae440ae15de0d7544cc3bb8a69f2e2e4589501 WatchSource:0}: Error finding container b72fd09554904f2217741930c1ae440ae15de0d7544cc3bb8a69f2e2e4589501: Status 404 returned error can't find the container with id b72fd09554904f2217741930c1ae440ae15de0d7544cc3bb8a69f2e2e4589501
Jul 08 23:23:58.521692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:58.521663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7464f4d4d-zzkmx" event={"ID":"9b95ca82-e911-482a-ae68-d49d2c641dc1","Type":"ContainerStarted","Data":"b72fd09554904f2217741930c1ae440ae15de0d7544cc3bb8a69f2e2e4589501"}
Jul 08 23:23:59.527114 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:59.527071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7464f4d4d-zzkmx" event={"ID":"9b95ca82-e911-482a-ae68-d49d2c641dc1","Type":"ContainerStarted","Data":"7f99f11c7893e0664008ed3ef979985a01c2d7cb30f83b6b30076bc459112661"}
Jul 08 23:23:59.527468 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:59.527146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:23:59.543747 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:23:59.543660    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7464f4d4d-zzkmx" podStartSLOduration=1.141695199 podStartE2EDuration="1.543644361s" totalImagesPullingTime="401.949162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:23:59.542680851 +0000 UTC m=+988.534514159" watchObservedRunningTime="2026-07-08 23:23:59.543644361 +0000 UTC m=+988.535477668"
Jul 08 23:24:10.537662 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:24:10.537633    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7464f4d4d-zzkmx"
Jul 08 23:35:57.587488 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.587460    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"]
Jul 08 23:35:57.590563 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.590547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.593820 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.593800    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e8178fd3-serving-cert\""
Jul 08 23:35:57.600579 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.600558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"]
Jul 08 23:35:57.692166 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.692142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4vh\" (UniqueName: \"kubernetes.io/projected/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-kube-api-access-dl4vh\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.692260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.692176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-openshift-service-ca\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.692260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.692203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.793546 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.793523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4vh\" (UniqueName: \"kubernetes.io/projected/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-kube-api-access-dl4vh\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.793647 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.793558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-openshift-service-ca\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.793647 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.793582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.793741 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:35:57.793726    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-serving-cert: secret "maas-api-e2e-ait-e8178fd3-serving-cert" not found
Jul 08 23:35:57.793789 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:35:57.793779    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls podName:b8dbb26a-8de6-4e0d-88de-bc070a21ac1f nodeName:}" failed. No retries permitted until 2026-07-08 23:35:58.293765156 +0000 UTC m=+1707.285598439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls") pod "maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" (UID: "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f") : secret "maas-api-e2e-ait-e8178fd3-serving-cert" not found
Jul 08 23:35:57.794182 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.794165    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-openshift-service-ca\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:57.803677 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:57.803651    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4vh\" (UniqueName: \"kubernetes.io/projected/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-kube-api-access-dl4vh\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:58.297125 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:58.297083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:58.299499 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:58.299471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls\") pod \"maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:58.500405 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:58.500374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:58.628322 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:58.628297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"]
Jul 08 23:35:58.631027 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:35:58.630999    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8dbb26a_8de6_4e0d_88de_bc070a21ac1f.slice/crio-368aa8320e646547515b84034e3cdf05c74365dd281f5a3bb728e3da8a39b75e WatchSource:0}: Error finding container 368aa8320e646547515b84034e3cdf05c74365dd281f5a3bb728e3da8a39b75e: Status 404 returned error can't find the container with id 368aa8320e646547515b84034e3cdf05c74365dd281f5a3bb728e3da8a39b75e
Jul 08 23:35:58.632442 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:58.632419    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 23:35:58.755796 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:58.755758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" event={"ID":"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f","Type":"ContainerStarted","Data":"368aa8320e646547515b84034e3cdf05c74365dd281f5a3bb728e3da8a39b75e"}
Jul 08 23:35:59.761858 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:59.761824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" event={"ID":"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f","Type":"ContainerStarted","Data":"1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080"}
Jul 08 23:35:59.762260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:59.761971    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:35:59.799407 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:35:59.799364    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" podStartSLOduration=2.283723089 podStartE2EDuration="2.799350767s" totalImagesPullingTime="515.627678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:35:59.797881778 +0000 UTC m=+1708.789715081" watchObservedRunningTime="2026-07-08 23:35:59.799350767 +0000 UTC m=+1708.791184073"
Jul 08 23:36:02.999231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:02.999199    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"]
Jul 08 23:36:02.999744 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:02.999385    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" podUID="b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" containerName="maas-api" containerID="cri-o://1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080" gracePeriod=30
Jul 08 23:36:03.236868 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.236848    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:36:03.337168 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.337118    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-openshift-service-ca\") pod \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") "
Jul 08 23:36:03.337260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.337173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls\") pod \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") "
Jul 08 23:36:03.337260 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.337203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl4vh\" (UniqueName: \"kubernetes.io/projected/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-kube-api-access-dl4vh\") pod \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\" (UID: \"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f\") "
Jul 08 23:36:03.337432 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.337412    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" (UID: "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:36:03.339240 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.339214    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" (UID: "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:36:03.339328 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.339255    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-kube-api-access-dl4vh" (OuterVolumeSpecName: "kube-api-access-dl4vh") pod "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" (UID: "b8dbb26a-8de6-4e0d-88de-bc070a21ac1f"). InnerVolumeSpecName "kube-api-access-dl4vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:36:03.438107 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.438083    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:03.438107 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.438104    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:03.438240 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.438114    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4vh\" (UniqueName: \"kubernetes.io/projected/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f-kube-api-access-dl4vh\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:03.778817 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.778790    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" containerID="1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080" exitCode=0
Jul 08 23:36:03.778922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.778839    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"
Jul 08 23:36:03.778922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.778883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" event={"ID":"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f","Type":"ContainerDied","Data":"1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080"}
Jul 08 23:36:03.779004 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.778938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr" event={"ID":"b8dbb26a-8de6-4e0d-88de-bc070a21ac1f","Type":"ContainerDied","Data":"368aa8320e646547515b84034e3cdf05c74365dd281f5a3bb728e3da8a39b75e"}
Jul 08 23:36:03.779004 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.778962    2577 scope.go:117] "RemoveContainer" containerID="1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080"
Jul 08 23:36:03.787525 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.787498    2577 scope.go:117] "RemoveContainer" containerID="1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080"
Jul 08 23:36:03.787818 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:03.787799    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080\": container with ID starting with 1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080 not found: ID does not exist" containerID="1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080"
Jul 08 23:36:03.787901 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.787828    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080"} err="failed to get container status \"1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080\": rpc error: code = NotFound desc = could not find container \"1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080\": container with ID starting with 1e7feb5cac535389aacfb2d132aaf9a27a5902617bd4144eff83fe344d44f080 not found: ID does not exist"
Jul 08 23:36:03.794813 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.794789    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"]
Jul 08 23:36:03.799038 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:03.799018    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8178fd3-559f7b7d67-5g5sr"]
Jul 08 23:36:05.520129 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:05.520097    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" path="/var/lib/kubelet/pods/b8dbb26a-8de6-4e0d-88de-bc070a21ac1f/volumes"
Jul 08 23:36:10.104918 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.104885    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"]
Jul 08 23:36:10.105365 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.105218    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" containerName="maas-api"
Jul 08 23:36:10.105365 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.105229    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" containerName="maas-api"
Jul 08 23:36:10.105365 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.105290    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8dbb26a-8de6-4e0d-88de-bc070a21ac1f" containerName="maas-api"
Jul 08 23:36:10.109602 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.109582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.112096 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.112076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2443f2d9-serving-cert\""
Jul 08 23:36:10.118071 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.118050    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"]
Jul 08 23:36:10.194620 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.194592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdm9\" (UniqueName: \"kubernetes.io/projected/30489d71-0823-4b1e-89f9-94e47a691fdc-kube-api-access-vzdm9\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.194736 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.194660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.194736 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.194725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30489d71-0823-4b1e-89f9-94e47a691fdc-openshift-service-ca\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.295208 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.295182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.295310 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.295216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30489d71-0823-4b1e-89f9-94e47a691fdc-openshift-service-ca\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.295310 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.295269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdm9\" (UniqueName: \"kubernetes.io/projected/30489d71-0823-4b1e-89f9-94e47a691fdc-kube-api-access-vzdm9\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.295431 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:10.295343    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-serving-cert: secret "maas-api-e2e-ait-2443f2d9-serving-cert" not found
Jul 08 23:36:10.295431 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:10.295429    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls podName:30489d71-0823-4b1e-89f9-94e47a691fdc nodeName:}" failed. No retries permitted until 2026-07-08 23:36:10.795411809 +0000 UTC m=+1719.787245093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls") pod "maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" (UID: "30489d71-0823-4b1e-89f9-94e47a691fdc") : secret "maas-api-e2e-ait-2443f2d9-serving-cert" not found
Jul 08 23:36:10.295878 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.295860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30489d71-0823-4b1e-89f9-94e47a691fdc-openshift-service-ca\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.303776 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.303753    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdm9\" (UniqueName: \"kubernetes.io/projected/30489d71-0823-4b1e-89f9-94e47a691fdc-kube-api-access-vzdm9\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.800017 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.799965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:10.802785 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:10.802695    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls\") pod \"maas-api-e2e-ait-2443f2d9-598447d47c-45dl2\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:11.021952 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:11.021923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:11.155327 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:11.155304    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"]
Jul 08 23:36:11.157362 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:36:11.157327    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30489d71_0823_4b1e_89f9_94e47a691fdc.slice/crio-bfb8844fa7f7b2ff90463f8405a8b932114f3148c05005876db880edc51ee74d WatchSource:0}: Error finding container bfb8844fa7f7b2ff90463f8405a8b932114f3148c05005876db880edc51ee74d: Status 404 returned error can't find the container with id bfb8844fa7f7b2ff90463f8405a8b932114f3148c05005876db880edc51ee74d
Jul 08 23:36:11.816591 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:11.816511    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" event={"ID":"30489d71-0823-4b1e-89f9-94e47a691fdc","Type":"ContainerStarted","Data":"e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132"}
Jul 08 23:36:11.816591 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:11.816564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" event={"ID":"30489d71-0823-4b1e-89f9-94e47a691fdc","Type":"ContainerStarted","Data":"bfb8844fa7f7b2ff90463f8405a8b932114f3148c05005876db880edc51ee74d"}
Jul 08 23:36:11.816591 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:11.816584    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:11.833300 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:11.833253    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" podStartSLOduration=1.510440275 podStartE2EDuration="1.833238145s" totalImagesPullingTime="322.79787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:36:11.83243562 +0000 UTC m=+1720.824268923" watchObservedRunningTime="2026-07-08 23:36:11.833238145 +0000 UTC m=+1720.825071451"
Jul 08 23:36:15.441383 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.441344    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"]
Jul 08 23:36:15.441959 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.441617    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" podUID="30489d71-0823-4b1e-89f9-94e47a691fdc" containerName="maas-api" containerID="cri-o://e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132" gracePeriod=30
Jul 08 23:36:15.689544 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.689518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:15.737448 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.737398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls\") pod \"30489d71-0823-4b1e-89f9-94e47a691fdc\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") "
Jul 08 23:36:15.737448 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.737426    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30489d71-0823-4b1e-89f9-94e47a691fdc-openshift-service-ca\") pod \"30489d71-0823-4b1e-89f9-94e47a691fdc\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") "
Jul 08 23:36:15.737576 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.737465    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzdm9\" (UniqueName: \"kubernetes.io/projected/30489d71-0823-4b1e-89f9-94e47a691fdc-kube-api-access-vzdm9\") pod \"30489d71-0823-4b1e-89f9-94e47a691fdc\" (UID: \"30489d71-0823-4b1e-89f9-94e47a691fdc\") "
Jul 08 23:36:15.737790 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.737770    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30489d71-0823-4b1e-89f9-94e47a691fdc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "30489d71-0823-4b1e-89f9-94e47a691fdc" (UID: "30489d71-0823-4b1e-89f9-94e47a691fdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:36:15.739443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.739420    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "30489d71-0823-4b1e-89f9-94e47a691fdc" (UID: "30489d71-0823-4b1e-89f9-94e47a691fdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:36:15.739493 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.739467    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30489d71-0823-4b1e-89f9-94e47a691fdc-kube-api-access-vzdm9" (OuterVolumeSpecName: "kube-api-access-vzdm9") pod "30489d71-0823-4b1e-89f9-94e47a691fdc" (UID: "30489d71-0823-4b1e-89f9-94e47a691fdc"). InnerVolumeSpecName "kube-api-access-vzdm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:36:15.834857 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.834833    2577 generic.go:356] "Generic (PLEG): container finished" podID="30489d71-0823-4b1e-89f9-94e47a691fdc" containerID="e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132" exitCode=0
Jul 08 23:36:15.834954 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.834874    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"
Jul 08 23:36:15.834954 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.834921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" event={"ID":"30489d71-0823-4b1e-89f9-94e47a691fdc","Type":"ContainerDied","Data":"e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132"}
Jul 08 23:36:15.835028 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.834971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2" event={"ID":"30489d71-0823-4b1e-89f9-94e47a691fdc","Type":"ContainerDied","Data":"bfb8844fa7f7b2ff90463f8405a8b932114f3148c05005876db880edc51ee74d"}
Jul 08 23:36:15.835028 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.834996    2577 scope.go:117] "RemoveContainer" containerID="e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132"
Jul 08 23:36:15.838343 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.838320    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30489d71-0823-4b1e-89f9-94e47a691fdc-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:15.838456 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.838348    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30489d71-0823-4b1e-89f9-94e47a691fdc-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:15.838456 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.838362    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdm9\" (UniqueName: \"kubernetes.io/projected/30489d71-0823-4b1e-89f9-94e47a691fdc-kube-api-access-vzdm9\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:15.843921 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.843906    2577 scope.go:117] "RemoveContainer" containerID="e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132"
Jul 08 23:36:15.844178 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:15.844160    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132\": container with ID starting with e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132 not found: ID does not exist" containerID="e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132"
Jul 08 23:36:15.844237 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.844187    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132"} err="failed to get container status \"e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132\": rpc error: code = NotFound desc = could not find container \"e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132\": container with ID starting with e359f5fbbd923a2c2de51cde37454bd1e33ad15bc92f4460c7e531c05f07a132 not found: ID does not exist"
Jul 08 23:36:15.858942 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.858915    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"]
Jul 08 23:36:15.862638 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:15.862620    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2443f2d9-598447d47c-45dl2"]
Jul 08 23:36:17.526052 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:17.526017    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30489d71-0823-4b1e-89f9-94e47a691fdc" path="/var/lib/kubelet/pods/30489d71-0823-4b1e-89f9-94e47a691fdc/volumes"
Jul 08 23:36:28.434984 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.434945    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"]
Jul 08 23:36:28.435515 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.435493    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30489d71-0823-4b1e-89f9-94e47a691fdc" containerName="maas-api"
Jul 08 23:36:28.435577 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.435520    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="30489d71-0823-4b1e-89f9-94e47a691fdc" containerName="maas-api"
Jul 08 23:36:28.435643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.435629    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="30489d71-0823-4b1e-89f9-94e47a691fdc" containerName="maas-api"
Jul 08 23:36:28.439860 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.439839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.442172 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.442147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-197d7ae2-serving-cert\""
Jul 08 23:36:28.447008 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.446943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"]
Jul 08 23:36:28.524768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.524749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.524871 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.524790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-openshift-service-ca\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.524871 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.524854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwn7\" (UniqueName: \"kubernetes.io/projected/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-kube-api-access-bhwn7\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.625461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.625439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-openshift-service-ca\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.625554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.625487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwn7\" (UniqueName: \"kubernetes.io/projected/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-kube-api-access-bhwn7\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.625554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.625521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.625665 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:28.625625    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-serving-cert: secret "maas-api-e2e-derive-197d7ae2-serving-cert" not found
Jul 08 23:36:28.625740 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:28.625680    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls podName:4fe70303-9f15-49b9-98a7-f3c574cf4ef1 nodeName:}" failed. No retries permitted until 2026-07-08 23:36:29.125661545 +0000 UTC m=+1738.117494828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls") pod "maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" (UID: "4fe70303-9f15-49b9-98a7-f3c574cf4ef1") : secret "maas-api-e2e-derive-197d7ae2-serving-cert" not found
Jul 08 23:36:28.626038 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.626018    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-openshift-service-ca\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:28.635678 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:28.635659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwn7\" (UniqueName: \"kubernetes.io/projected/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-kube-api-access-bhwn7\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:29.130863 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:29.130830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:29.133164 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:29.133132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls\") pod \"maas-api-e2e-derive-197d7ae2-97c769dd6-68p95\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:29.351180 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:29.351147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:29.484235 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:29.484212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"]
Jul 08 23:36:29.486416 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:36:29.486392    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe70303_9f15_49b9_98a7_f3c574cf4ef1.slice/crio-52556cf2130abfb92976a6149049550c2ad2b30a1c093f33c1c4c55f64d30c4f WatchSource:0}: Error finding container 52556cf2130abfb92976a6149049550c2ad2b30a1c093f33c1c4c55f64d30c4f: Status 404 returned error can't find the container with id 52556cf2130abfb92976a6149049550c2ad2b30a1c093f33c1c4c55f64d30c4f
Jul 08 23:36:29.903238 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:29.903210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" event={"ID":"4fe70303-9f15-49b9-98a7-f3c574cf4ef1","Type":"ContainerStarted","Data":"52556cf2130abfb92976a6149049550c2ad2b30a1c093f33c1c4c55f64d30c4f"}
Jul 08 23:36:30.908613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:30.908578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" event={"ID":"4fe70303-9f15-49b9-98a7-f3c574cf4ef1","Type":"ContainerStarted","Data":"a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6"}
Jul 08 23:36:30.909041 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:30.908728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:30.926402 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:30.926361    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" podStartSLOduration=2.551353845 podStartE2EDuration="2.926347244s" totalImagesPullingTime="374.993399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:36:30.924768268 +0000 UTC m=+1739.916601574" watchObservedRunningTime="2026-07-08 23:36:30.926347244 +0000 UTC m=+1739.918180549"
Jul 08 23:36:33.119824 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.119793    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"]
Jul 08 23:36:33.120229 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.120018    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" podUID="4fe70303-9f15-49b9-98a7-f3c574cf4ef1" containerName="maas-api" containerID="cri-o://a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6" gracePeriod=30
Jul 08 23:36:33.362192 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.362171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:33.468329 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.468268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-openshift-service-ca\") pod \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") "
Jul 08 23:36:33.468329 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.468304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls\") pod \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") "
Jul 08 23:36:33.468329 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.468328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhwn7\" (UniqueName: \"kubernetes.io/projected/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-kube-api-access-bhwn7\") pod \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\" (UID: \"4fe70303-9f15-49b9-98a7-f3c574cf4ef1\") "
Jul 08 23:36:33.468686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.468651    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4fe70303-9f15-49b9-98a7-f3c574cf4ef1" (UID: "4fe70303-9f15-49b9-98a7-f3c574cf4ef1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:36:33.470299 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.470268    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4fe70303-9f15-49b9-98a7-f3c574cf4ef1" (UID: "4fe70303-9f15-49b9-98a7-f3c574cf4ef1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:36:33.470381 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.470307    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-kube-api-access-bhwn7" (OuterVolumeSpecName: "kube-api-access-bhwn7") pod "4fe70303-9f15-49b9-98a7-f3c574cf4ef1" (UID: "4fe70303-9f15-49b9-98a7-f3c574cf4ef1"). InnerVolumeSpecName "kube-api-access-bhwn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:36:33.569443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.569412    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:33.569443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.569436    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:33.569608 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.569449    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhwn7\" (UniqueName: \"kubernetes.io/projected/4fe70303-9f15-49b9-98a7-f3c574cf4ef1-kube-api-access-bhwn7\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:36:33.922288 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.922258    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fe70303-9f15-49b9-98a7-f3c574cf4ef1" containerID="a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6" exitCode=0
Jul 08 23:36:33.922403 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.922305    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"
Jul 08 23:36:33.922403 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.922308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" event={"ID":"4fe70303-9f15-49b9-98a7-f3c574cf4ef1","Type":"ContainerDied","Data":"a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6"}
Jul 08 23:36:33.922403 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.922381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95" event={"ID":"4fe70303-9f15-49b9-98a7-f3c574cf4ef1","Type":"ContainerDied","Data":"52556cf2130abfb92976a6149049550c2ad2b30a1c093f33c1c4c55f64d30c4f"}
Jul 08 23:36:33.922509 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.922406    2577 scope.go:117] "RemoveContainer" containerID="a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6"
Jul 08 23:36:33.930762 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.930741    2577 scope.go:117] "RemoveContainer" containerID="a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6"
Jul 08 23:36:33.931007 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:36:33.930988    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6\": container with ID starting with a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6 not found: ID does not exist" containerID="a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6"
Jul 08 23:36:33.931073 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.931013    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6"} err="failed to get container status \"a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6\": rpc error: code = NotFound desc = could not find container \"a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6\": container with ID starting with a635c877ad66598162813c9ad7b8a9833b70306c6fcc695bd658ea0515e364a6 not found: ID does not exist"
Jul 08 23:36:33.939167 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.939146    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"]
Jul 08 23:36:33.947448 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:33.947420    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-197d7ae2-97c769dd6-68p95"]
Jul 08 23:36:35.520167 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:36:35.520136    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe70303-9f15-49b9-98a7-f3c574cf4ef1" path="/var/lib/kubelet/pods/4fe70303-9f15-49b9-98a7-f3c574cf4ef1/volumes"
Jul 08 23:38:16.875270 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.875237    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"]
Jul 08 23:38:16.875731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.875580    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe70303-9f15-49b9-98a7-f3c574cf4ef1" containerName="maas-api"
Jul 08 23:38:16.875731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.875591    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe70303-9f15-49b9-98a7-f3c574cf4ef1" containerName="maas-api"
Jul 08 23:38:16.875731 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.875652    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe70303-9f15-49b9-98a7-f3c574cf4ef1" containerName="maas-api"
Jul 08 23:38:16.878677 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.878656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:16.881948 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.881926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f40dc7f6-serving-cert\""
Jul 08 23:38:16.889558 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.889534    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"]
Jul 08 23:38:16.935489 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.935465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7b5\" (UniqueName: \"kubernetes.io/projected/f40cbfe0-0c95-444a-971a-a865ea9de9e8-kube-api-access-mq7b5\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:16.935595 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.935504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:16.935595 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:16.935527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f40cbfe0-0c95-444a-971a-a865ea9de9e8-openshift-service-ca\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.036721 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.036676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.036830 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.036727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f40cbfe0-0c95-444a-971a-a865ea9de9e8-openshift-service-ca\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.036830 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.036781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7b5\" (UniqueName: \"kubernetes.io/projected/f40cbfe0-0c95-444a-971a-a865ea9de9e8-kube-api-access-mq7b5\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.036830 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:38:17.036813    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-serving-cert: secret "maas-api-e2e-mt-f40dc7f6-serving-cert" not found
Jul 08 23:38:17.036984 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:38:17.036870    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls podName:f40cbfe0-0c95-444a-971a-a865ea9de9e8 nodeName:}" failed. No retries permitted until 2026-07-08 23:38:17.536855802 +0000 UTC m=+1846.528689085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls") pod "maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" (UID: "f40cbfe0-0c95-444a-971a-a865ea9de9e8") : secret "maas-api-e2e-mt-f40dc7f6-serving-cert" not found
Jul 08 23:38:17.037415 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.037390    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f40cbfe0-0c95-444a-971a-a865ea9de9e8-openshift-service-ca\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.045925 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.045895    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7b5\" (UniqueName: \"kubernetes.io/projected/f40cbfe0-0c95-444a-971a-a865ea9de9e8-kube-api-access-mq7b5\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.540692 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.540664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.543043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.543016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls\") pod \"maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.789822 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.789793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:17.919668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:17.919635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"]
Jul 08 23:38:17.920978 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:38:17.920955    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40cbfe0_0c95_444a_971a_a865ea9de9e8.slice/crio-efba247d6e2df88b654f9ee18d18319e3f7c7a3f96e982aa46b01187c02b0d35 WatchSource:0}: Error finding container efba247d6e2df88b654f9ee18d18319e3f7c7a3f96e982aa46b01187c02b0d35: Status 404 returned error can't find the container with id efba247d6e2df88b654f9ee18d18319e3f7c7a3f96e982aa46b01187c02b0d35
Jul 08 23:38:18.406941 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:18.406916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" event={"ID":"f40cbfe0-0c95-444a-971a-a865ea9de9e8","Type":"ContainerStarted","Data":"9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c"}
Jul 08 23:38:18.407057 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:18.406971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" event={"ID":"f40cbfe0-0c95-444a-971a-a865ea9de9e8","Type":"ContainerStarted","Data":"efba247d6e2df88b654f9ee18d18319e3f7c7a3f96e982aa46b01187c02b0d35"}
Jul 08 23:38:18.407057 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:18.407003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:18.423865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:18.423824    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" podStartSLOduration=2.028493288 podStartE2EDuration="2.423809108s" totalImagesPullingTime="395.31582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:38:18.422511135 +0000 UTC m=+1847.414344442" watchObservedRunningTime="2026-07-08 23:38:18.423809108 +0000 UTC m=+1847.415642413"
Jul 08 23:38:24.416755 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:24.416721    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:28.648029 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:28.647994    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"]
Jul 08 23:38:28.648463 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:28.648206    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" podUID="f40cbfe0-0c95-444a-971a-a865ea9de9e8" containerName="maas-api" containerID="cri-o://9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c" gracePeriod=30
Jul 08 23:38:28.905188 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:28.905133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:29.020173 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.020149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f40cbfe0-0c95-444a-971a-a865ea9de9e8-openshift-service-ca\") pod \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") "
Jul 08 23:38:29.020302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.020188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq7b5\" (UniqueName: \"kubernetes.io/projected/f40cbfe0-0c95-444a-971a-a865ea9de9e8-kube-api-access-mq7b5\") pod \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") "
Jul 08 23:38:29.020302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.020255    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls\") pod \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\" (UID: \"f40cbfe0-0c95-444a-971a-a865ea9de9e8\") "
Jul 08 23:38:29.020537 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.020516    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40cbfe0-0c95-444a-971a-a865ea9de9e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f40cbfe0-0c95-444a-971a-a865ea9de9e8" (UID: "f40cbfe0-0c95-444a-971a-a865ea9de9e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:38:29.022263 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.022242    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f40cbfe0-0c95-444a-971a-a865ea9de9e8" (UID: "f40cbfe0-0c95-444a-971a-a865ea9de9e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:38:29.022349 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.022324    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40cbfe0-0c95-444a-971a-a865ea9de9e8-kube-api-access-mq7b5" (OuterVolumeSpecName: "kube-api-access-mq7b5") pod "f40cbfe0-0c95-444a-971a-a865ea9de9e8" (UID: "f40cbfe0-0c95-444a-971a-a865ea9de9e8"). InnerVolumeSpecName "kube-api-access-mq7b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:38:29.121709 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.121686    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7b5\" (UniqueName: \"kubernetes.io/projected/f40cbfe0-0c95-444a-971a-a865ea9de9e8-kube-api-access-mq7b5\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:38:29.121804 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.121726    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f40cbfe0-0c95-444a-971a-a865ea9de9e8-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:38:29.121804 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.121736    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f40cbfe0-0c95-444a-971a-a865ea9de9e8-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:38:29.456570 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.456533    2577 generic.go:356] "Generic (PLEG): container finished" podID="f40cbfe0-0c95-444a-971a-a865ea9de9e8" containerID="9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c" exitCode=0
Jul 08 23:38:29.456748 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.456585    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"
Jul 08 23:38:29.456748 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.456593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" event={"ID":"f40cbfe0-0c95-444a-971a-a865ea9de9e8","Type":"ContainerDied","Data":"9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c"}
Jul 08 23:38:29.456748 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.456684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz" event={"ID":"f40cbfe0-0c95-444a-971a-a865ea9de9e8","Type":"ContainerDied","Data":"efba247d6e2df88b654f9ee18d18319e3f7c7a3f96e982aa46b01187c02b0d35"}
Jul 08 23:38:29.456748 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.456740    2577 scope.go:117] "RemoveContainer" containerID="9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c"
Jul 08 23:38:29.466535 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.466519    2577 scope.go:117] "RemoveContainer" containerID="9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c"
Jul 08 23:38:29.466817 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:38:29.466793    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c\": container with ID starting with 9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c not found: ID does not exist" containerID="9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c"
Jul 08 23:38:29.466876 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.466826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c"} err="failed to get container status \"9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c\": rpc error: code = NotFound desc = could not find container \"9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c\": container with ID starting with 9f3b2042aba7665d0b213681698237cdaa8059d2da765ddfffdc6dc42066693c not found: ID does not exist"
Jul 08 23:38:29.478302 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.478278    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"]
Jul 08 23:38:29.484056 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.484033    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f40dc7f6-69747b778b-8krpz"]
Jul 08 23:38:29.519539 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:29.519519    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40cbfe0-0c95-444a-971a-a865ea9de9e8" path="/var/lib/kubelet/pods/f40cbfe0-0c95-444a-971a-a865ea9de9e8/volumes"
Jul 08 23:38:56.996712 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:56.996666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"]
Jul 08 23:38:56.997244 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:56.997026    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40cbfe0-0c95-444a-971a-a865ea9de9e8" containerName="maas-api"
Jul 08 23:38:56.997244 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:56.997039    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40cbfe0-0c95-444a-971a-a865ea9de9e8" containerName="maas-api"
Jul 08 23:38:56.997244 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:56.997110    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40cbfe0-0c95-444a-971a-a865ea9de9e8" containerName="maas-api"
Jul 08 23:38:56.999865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:56.999846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.002176 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.002153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-58c9c277-serving-cert\""
Jul 08 23:38:57.009636 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.009617    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"]
Jul 08 23:38:57.138589 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.138566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzq58\" (UniqueName: \"kubernetes.io/projected/138cf7f8-892e-47ec-ab17-993a661d0d26-kube-api-access-pzq58\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.138695 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.138630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138cf7f8-892e-47ec-ab17-993a661d0d26-maas-api-tls\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.138695 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.138654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138cf7f8-892e-47ec-ab17-993a661d0d26-openshift-service-ca\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.239299 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.239276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138cf7f8-892e-47ec-ab17-993a661d0d26-maas-api-tls\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.239396 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.239315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138cf7f8-892e-47ec-ab17-993a661d0d26-openshift-service-ca\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.239396 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.239359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzq58\" (UniqueName: \"kubernetes.io/projected/138cf7f8-892e-47ec-ab17-993a661d0d26-kube-api-access-pzq58\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.240154 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.240129    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138cf7f8-892e-47ec-ab17-993a661d0d26-openshift-service-ca\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.241766 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.241738    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138cf7f8-892e-47ec-ab17-993a661d0d26-maas-api-tls\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.249201 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.249137    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzq58\" (UniqueName: \"kubernetes.io/projected/138cf7f8-892e-47ec-ab17-993a661d0d26-kube-api-access-pzq58\") pod \"maas-api-e2e-mt-58c9c277-755f49996d-sx6fk\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.310613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.310590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:57.440989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.440959    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"]
Jul 08 23:38:57.443460 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:38:57.443437    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138cf7f8_892e_47ec_ab17_993a661d0d26.slice/crio-b1d0236332b8bfd473115640e36588ea3926f1d524cfba722af3f3b6046ca918 WatchSource:0}: Error finding container b1d0236332b8bfd473115640e36588ea3926f1d524cfba722af3f3b6046ca918: Status 404 returned error can't find the container with id b1d0236332b8bfd473115640e36588ea3926f1d524cfba722af3f3b6046ca918
Jul 08 23:38:57.586257 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:57.586189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk" event={"ID":"138cf7f8-892e-47ec-ab17-993a661d0d26","Type":"ContainerStarted","Data":"b1d0236332b8bfd473115640e36588ea3926f1d524cfba722af3f3b6046ca918"}
Jul 08 23:38:58.592375 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:58.592341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk" event={"ID":"138cf7f8-892e-47ec-ab17-993a661d0d26","Type":"ContainerStarted","Data":"d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107"}
Jul 08 23:38:58.592745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:58.592399    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:38:58.608791 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:38:58.608748    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk" podStartSLOduration=2.113962582 podStartE2EDuration="2.608731617s" totalImagesPullingTime="494.769035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:38:58.607555283 +0000 UTC m=+1887.599388602" watchObservedRunningTime="2026-07-08 23:38:58.608731617 +0000 UTC m=+1887.600564942"
Jul 08 23:39:04.601663 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:04.601632    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:39:29.313955 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.313918    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"]
Jul 08 23:39:29.314451 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.314187    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk" podUID="138cf7f8-892e-47ec-ab17-993a661d0d26" containerName="maas-api" containerID="cri-o://d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107" gracePeriod=30
Jul 08 23:39:29.553782 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.553757    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:39:29.677083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.677057    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138cf7f8-892e-47ec-ab17-993a661d0d26-openshift-service-ca\") pod \"138cf7f8-892e-47ec-ab17-993a661d0d26\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") "
Jul 08 23:39:29.677223 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.677091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzq58\" (UniqueName: \"kubernetes.io/projected/138cf7f8-892e-47ec-ab17-993a661d0d26-kube-api-access-pzq58\") pod \"138cf7f8-892e-47ec-ab17-993a661d0d26\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") "
Jul 08 23:39:29.677223 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.677138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138cf7f8-892e-47ec-ab17-993a661d0d26-maas-api-tls\") pod \"138cf7f8-892e-47ec-ab17-993a661d0d26\" (UID: \"138cf7f8-892e-47ec-ab17-993a661d0d26\") "
Jul 08 23:39:29.677449 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.677426    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138cf7f8-892e-47ec-ab17-993a661d0d26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "138cf7f8-892e-47ec-ab17-993a661d0d26" (UID: "138cf7f8-892e-47ec-ab17-993a661d0d26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:39:29.679268 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.679245    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138cf7f8-892e-47ec-ab17-993a661d0d26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "138cf7f8-892e-47ec-ab17-993a661d0d26" (UID: "138cf7f8-892e-47ec-ab17-993a661d0d26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:39:29.679350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.679263    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138cf7f8-892e-47ec-ab17-993a661d0d26-kube-api-access-pzq58" (OuterVolumeSpecName: "kube-api-access-pzq58") pod "138cf7f8-892e-47ec-ab17-993a661d0d26" (UID: "138cf7f8-892e-47ec-ab17-993a661d0d26"). InnerVolumeSpecName "kube-api-access-pzq58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:39:29.731959 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.731937    2577 generic.go:356] "Generic (PLEG): container finished" podID="138cf7f8-892e-47ec-ab17-993a661d0d26" containerID="d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107" exitCode=0
Jul 08 23:39:29.732040 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.731983    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"
Jul 08 23:39:29.732040 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.732008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk" event={"ID":"138cf7f8-892e-47ec-ab17-993a661d0d26","Type":"ContainerDied","Data":"d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107"}
Jul 08 23:39:29.732113 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.732050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk" event={"ID":"138cf7f8-892e-47ec-ab17-993a661d0d26","Type":"ContainerDied","Data":"b1d0236332b8bfd473115640e36588ea3926f1d524cfba722af3f3b6046ca918"}
Jul 08 23:39:29.732113 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.732073    2577 scope.go:117] "RemoveContainer" containerID="d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107"
Jul 08 23:39:29.740327 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.740308    2577 scope.go:117] "RemoveContainer" containerID="d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107"
Jul 08 23:39:29.740559 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:39:29.740540    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107\": container with ID starting with d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107 not found: ID does not exist" containerID="d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107"
Jul 08 23:39:29.740607 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.740568    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107"} err="failed to get container status \"d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107\": rpc error: code = NotFound desc = could not find container \"d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107\": container with ID starting with d43ae308b232a9b5b65ee76634042bb03a955706f6a0839521679fee53935107 not found: ID does not exist"
Jul 08 23:39:29.751649 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.751630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"]
Jul 08 23:39:29.755421 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.755402    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-58c9c277-755f49996d-sx6fk"]
Jul 08 23:39:29.778159 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.778135    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/138cf7f8-892e-47ec-ab17-993a661d0d26-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:39:29.778159 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.778157    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzq58\" (UniqueName: \"kubernetes.io/projected/138cf7f8-892e-47ec-ab17-993a661d0d26-kube-api-access-pzq58\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:39:29.778267 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:29.778168    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/138cf7f8-892e-47ec-ab17-993a661d0d26-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:39:31.520194 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:39:31.520162    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="138cf7f8-892e-47ec-ab17-993a661d0d26" path="/var/lib/kubelet/pods/138cf7f8-892e-47ec-ab17-993a661d0d26/volumes"
Jul 08 23:41:19.863926 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.863895    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"]
Jul 08 23:41:19.866148 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.864214    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="138cf7f8-892e-47ec-ab17-993a661d0d26" containerName="maas-api"
Jul 08 23:41:19.866148 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.864226    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="138cf7f8-892e-47ec-ab17-993a661d0d26" containerName="maas-api"
Jul 08 23:41:19.866148 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.864319    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="138cf7f8-892e-47ec-ab17-993a661d0d26" containerName="maas-api"
Jul 08 23:41:19.867007 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.866990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:19.869339 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.869316    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-08d361-serving-cert\""
Jul 08 23:41:19.878876 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.878851    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"]
Jul 08 23:41:19.981167 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.981133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmqrd\" (UniqueName: \"kubernetes.io/projected/0469a3f1-3f20-4776-b353-b7a55ce83032-kube-api-access-lmqrd\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:19.981320 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.981183    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469a3f1-3f20-4776-b353-b7a55ce83032-maas-api-tls\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:19.981320 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:19.981212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469a3f1-3f20-4776-b353-b7a55ce83032-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.082503 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.082478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmqrd\" (UniqueName: \"kubernetes.io/projected/0469a3f1-3f20-4776-b353-b7a55ce83032-kube-api-access-lmqrd\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.082618 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.082515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469a3f1-3f20-4776-b353-b7a55ce83032-maas-api-tls\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.082618 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.082539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469a3f1-3f20-4776-b353-b7a55ce83032-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.083137 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.083120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469a3f1-3f20-4776-b353-b7a55ce83032-openshift-service-ca\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.085046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.085029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469a3f1-3f20-4776-b353-b7a55ce83032-maas-api-tls\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.091131 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.091097    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmqrd\" (UniqueName: \"kubernetes.io/projected/0469a3f1-3f20-4776-b353-b7a55ce83032-kube-api-access-lmqrd\") pod \"maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.176516 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.176463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:20.308348 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.308322    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"]
Jul 08 23:41:20.310448 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:41:20.310420    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0469a3f1_3f20_4776_b353_b7a55ce83032.slice/crio-2c499da4ed19b71b095894c535e0ed023047d149fc1439cc49b8d042bdd476f8 WatchSource:0}: Error finding container 2c499da4ed19b71b095894c535e0ed023047d149fc1439cc49b8d042bdd476f8: Status 404 returned error can't find the container with id 2c499da4ed19b71b095894c535e0ed023047d149fc1439cc49b8d042bdd476f8
Jul 08 23:41:20.311771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:20.311749    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 23:41:21.228495 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:21.228463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc" event={"ID":"0469a3f1-3f20-4776-b353-b7a55ce83032","Type":"ContainerStarted","Data":"302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0"}
Jul 08 23:41:21.228883 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:21.228521    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:21.228883 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:21.228539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc" event={"ID":"0469a3f1-3f20-4776-b353-b7a55ce83032","Type":"ContainerStarted","Data":"2c499da4ed19b71b095894c535e0ed023047d149fc1439cc49b8d042bdd476f8"}
Jul 08 23:41:21.249446 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:21.249403    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc" podStartSLOduration=1.735328223 podStartE2EDuration="2.2493865s" totalImagesPullingTime="514.058277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:41:21.248516947 +0000 UTC m=+2030.240350253" watchObservedRunningTime="2026-07-08 23:41:21.2493865 +0000 UTC m=+2030.241219854"
Jul 08 23:41:27.237934 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:27.237907    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:41:27.892548 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:27.892515    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"]
Jul 08 23:41:27.896220 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:27.896199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:27.898661 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:27.898640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-717c2b-serving-cert\""
Jul 08 23:41:27.905970 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:27.905946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"]
Jul 08 23:41:28.048366 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.048334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2z7\" (UniqueName: \"kubernetes.io/projected/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-kube-api-access-5c2z7\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.048483 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.048398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.048483 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.048473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-openshift-service-ca\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.149577 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.149516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.149577 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.149567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-openshift-service-ca\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.149725 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:41:28.149679    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-serving-cert: secret "maas-api-e2e-inf-b-717c2b-serving-cert" not found
Jul 08 23:41:28.149769 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.149751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2z7\" (UniqueName: \"kubernetes.io/projected/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-kube-api-access-5c2z7\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.149804 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:41:28.149785    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls podName:f6048b80-5d4f-44f9-85b5-4ce6bcd1e221 nodeName:}" failed. No retries permitted until 2026-07-08 23:41:28.649760895 +0000 UTC m=+2037.641594184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls") pod "maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" (UID: "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221") : secret "maas-api-e2e-inf-b-717c2b-serving-cert" not found
Jul 08 23:41:28.150164 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.150146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-openshift-service-ca\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.160231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.160203    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2z7\" (UniqueName: \"kubernetes.io/projected/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-kube-api-access-5c2z7\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.653643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.653618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.655957 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.655938    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls\") pod \"maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.808399 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.808371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:28.950855 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:28.950739    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"]
Jul 08 23:41:29.265265 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:29.265185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" event={"ID":"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221","Type":"ContainerStarted","Data":"0da5f53071aaa5ce85f47195d1c0a4c7c64014fc41a77e3cc208f5bf8d2d1127"}
Jul 08 23:41:30.271335 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:30.271294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" event={"ID":"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221","Type":"ContainerStarted","Data":"157b8f4c24fb81bbda5cfa94d4783b3f2ab659eac8fde63e453b7f3ca3704a44"}
Jul 08 23:41:30.271725 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:30.271362    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:30.287710 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:30.287669    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" podStartSLOduration=2.90203667 podStartE2EDuration="3.287655848s" totalImagesPullingTime="385.619178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:41:30.286979271 +0000 UTC m=+2039.278812577" watchObservedRunningTime="2026-07-08 23:41:30.287655848 +0000 UTC m=+2039.279489153"
Jul 08 23:41:36.281291 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:36.281262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:41:39.734584 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.734548    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"]
Jul 08 23:41:39.737990 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.737973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.740326 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.740301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-08d361\"/\"kube-root-ca.crt\""
Jul 08 23:41:39.741300 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.741279    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-08d361\"/\"default-dockercfg-nw8d5\""
Jul 08 23:41:39.741370 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.741325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-08d361\"/\"openshift-service-ca.crt\""
Jul 08 23:41:39.741414 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.741371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-08d361\"/\"test-model-08d361-kserve-self-signed-certs\""
Jul 08 23:41:39.747475 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.747451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"]
Jul 08 23:41:39.836634 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.836612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-model-cache\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.836768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.836643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtmf\" (UniqueName: \"kubernetes.io/projected/32c40ca4-c612-4290-9f2c-94053f8d2458-kube-api-access-ggtmf\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.836768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.836668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-home\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.836768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.836745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-tmp-dir\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.836902 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.836808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-dshm\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.836902 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.836826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c40ca4-c612-4290-9f2c-94053f8d2458-tls-certs\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.937891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.937856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-model-cache\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.937905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtmf\" (UniqueName: \"kubernetes.io/projected/32c40ca4-c612-4290-9f2c-94053f8d2458-kube-api-access-ggtmf\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.937940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-home\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.937984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-tmp-dir\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938159 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.938051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-dshm\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938159 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.938073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c40ca4-c612-4290-9f2c-94053f8d2458-tls-certs\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938298 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.938278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-model-cache\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938357 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.938333    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-home\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.938417 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.938359    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-tmp-dir\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.940176 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.940153    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-dshm\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.940437 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.940419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c40ca4-c612-4290-9f2c-94053f8d2458-tls-certs\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:39.945461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:39.945445    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtmf\" (UniqueName: \"kubernetes.io/projected/32c40ca4-c612-4290-9f2c-94053f8d2458-kube-api-access-ggtmf\") pod \"test-model-08d361-kserve-7dcb84ffd5-mplqc\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") " pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:40.048693 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:40.048635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:40.175970 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:40.175908    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"]
Jul 08 23:41:40.178282 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:41:40.178262    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32c40ca4_c612_4290_9f2c_94053f8d2458.slice/crio-d85c77971eb622f5d692e8414c0a4d9361db69b5a17b8c4cc2ae042cf1b22f69 WatchSource:0}: Error finding container d85c77971eb622f5d692e8414c0a4d9361db69b5a17b8c4cc2ae042cf1b22f69: Status 404 returned error can't find the container with id d85c77971eb622f5d692e8414c0a4d9361db69b5a17b8c4cc2ae042cf1b22f69
Jul 08 23:41:40.320425 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:40.320344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc" event={"ID":"32c40ca4-c612-4290-9f2c-94053f8d2458","Type":"ContainerStarted","Data":"057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d"}
Jul 08 23:41:40.320425 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:40.320392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc" event={"ID":"32c40ca4-c612-4290-9f2c-94053f8d2458","Type":"ContainerStarted","Data":"d85c77971eb622f5d692e8414c0a4d9361db69b5a17b8c4cc2ae042cf1b22f69"}
Jul 08 23:41:40.337624 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:40.337579    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc" podStartSLOduration=1.337563233 podStartE2EDuration="1.337563233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:41:40.336489831 +0000 UTC m=+2049.328323138" watchObservedRunningTime="2026-07-08 23:41:40.337563233 +0000 UTC m=+2049.329396539"
Jul 08 23:41:50.049218 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:50.049188    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:50.049218 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:50.049228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:50.061333 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:50.061313    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:41:50.379966 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:41:50.379943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:42:35.904611 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:35.904572    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"]
Jul 08 23:42:37.639471 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:37.639436    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"]
Jul 08 23:42:37.639925 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:37.639682    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc" podUID="0469a3f1-3f20-4776-b353-b7a55ce83032" containerName="maas-api" containerID="cri-o://302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0" gracePeriod=30
Jul 08 23:42:38.080771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.080741    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:42:38.188989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.188929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469a3f1-3f20-4776-b353-b7a55ce83032-maas-api-tls\") pod \"0469a3f1-3f20-4776-b353-b7a55ce83032\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") "
Jul 08 23:42:38.188989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.188982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469a3f1-3f20-4776-b353-b7a55ce83032-openshift-service-ca\") pod \"0469a3f1-3f20-4776-b353-b7a55ce83032\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") "
Jul 08 23:42:38.189132 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.189009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmqrd\" (UniqueName: \"kubernetes.io/projected/0469a3f1-3f20-4776-b353-b7a55ce83032-kube-api-access-lmqrd\") pod \"0469a3f1-3f20-4776-b353-b7a55ce83032\" (UID: \"0469a3f1-3f20-4776-b353-b7a55ce83032\") "
Jul 08 23:42:38.189386 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.189352    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0469a3f1-3f20-4776-b353-b7a55ce83032-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0469a3f1-3f20-4776-b353-b7a55ce83032" (UID: "0469a3f1-3f20-4776-b353-b7a55ce83032"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:42:38.191038 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.191016    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0469a3f1-3f20-4776-b353-b7a55ce83032-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0469a3f1-3f20-4776-b353-b7a55ce83032" (UID: "0469a3f1-3f20-4776-b353-b7a55ce83032"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:42:38.191121 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.191068    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0469a3f1-3f20-4776-b353-b7a55ce83032-kube-api-access-lmqrd" (OuterVolumeSpecName: "kube-api-access-lmqrd") pod "0469a3f1-3f20-4776-b353-b7a55ce83032" (UID: "0469a3f1-3f20-4776-b353-b7a55ce83032"). InnerVolumeSpecName "kube-api-access-lmqrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:42:38.290517 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.290490    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0469a3f1-3f20-4776-b353-b7a55ce83032-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:38.290621 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.290541    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0469a3f1-3f20-4776-b353-b7a55ce83032-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:38.290621 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.290559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmqrd\" (UniqueName: \"kubernetes.io/projected/0469a3f1-3f20-4776-b353-b7a55ce83032-kube-api-access-lmqrd\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:38.591720 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.591632    2577 generic.go:356] "Generic (PLEG): container finished" podID="0469a3f1-3f20-4776-b353-b7a55ce83032" containerID="302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0" exitCode=0
Jul 08 23:42:38.591720 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.591684    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"
Jul 08 23:42:38.591872 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.591734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc" event={"ID":"0469a3f1-3f20-4776-b353-b7a55ce83032","Type":"ContainerDied","Data":"302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0"}
Jul 08 23:42:38.591872 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.591791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc" event={"ID":"0469a3f1-3f20-4776-b353-b7a55ce83032","Type":"ContainerDied","Data":"2c499da4ed19b71b095894c535e0ed023047d149fc1439cc49b8d042bdd476f8"}
Jul 08 23:42:38.591872 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.591815    2577 scope.go:117] "RemoveContainer" containerID="302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0"
Jul 08 23:42:38.600511 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.600494    2577 scope.go:117] "RemoveContainer" containerID="302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0"
Jul 08 23:42:38.600786 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:42:38.600758    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0\": container with ID starting with 302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0 not found: ID does not exist" containerID="302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0"
Jul 08 23:42:38.600881 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.600794    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0"} err="failed to get container status \"302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0\": rpc error: code = NotFound desc = could not find container \"302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0\": container with ID starting with 302a51fea3eed6dcc3379b9e39ef6354731ad456e643a721895dace4760e22d0 not found: ID does not exist"
Jul 08 23:42:38.612297 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.612274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"]
Jul 08 23:42:38.613875 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:38.613851    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-08d361-58bc65fbc8-rw9mc"]
Jul 08 23:42:39.520838 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:39.520799    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0469a3f1-3f20-4776-b353-b7a55ce83032" path="/var/lib/kubelet/pods/0469a3f1-3f20-4776-b353-b7a55ce83032/volumes"
Jul 08 23:42:50.924981 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:50.924942    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc" podUID="32c40ca4-c612-4290-9f2c-94053f8d2458" containerName="main" containerID="cri-o://057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d" gracePeriod=45
Jul 08 23:42:51.175889 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.175833    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:42:51.289031 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289006    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-model-cache\") pod \"32c40ca4-c612-4290-9f2c-94053f8d2458\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") "
Jul 08 23:42:51.289160 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-dshm\") pod \"32c40ca4-c612-4290-9f2c-94053f8d2458\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") "
Jul 08 23:42:51.289160 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-home\") pod \"32c40ca4-c612-4290-9f2c-94053f8d2458\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") "
Jul 08 23:42:51.289160 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c40ca4-c612-4290-9f2c-94053f8d2458-tls-certs\") pod \"32c40ca4-c612-4290-9f2c-94053f8d2458\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") "
Jul 08 23:42:51.289329 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289169    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggtmf\" (UniqueName: \"kubernetes.io/projected/32c40ca4-c612-4290-9f2c-94053f8d2458-kube-api-access-ggtmf\") pod \"32c40ca4-c612-4290-9f2c-94053f8d2458\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") "
Jul 08 23:42:51.289329 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289222    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-tmp-dir\") pod \"32c40ca4-c612-4290-9f2c-94053f8d2458\" (UID: \"32c40ca4-c612-4290-9f2c-94053f8d2458\") "
Jul 08 23:42:51.289329 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289238    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-model-cache" (OuterVolumeSpecName: "model-cache") pod "32c40ca4-c612-4290-9f2c-94053f8d2458" (UID: "32c40ca4-c612-4290-9f2c-94053f8d2458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:42:51.289486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289372    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-home" (OuterVolumeSpecName: "home") pod "32c40ca4-c612-4290-9f2c-94053f8d2458" (UID: "32c40ca4-c612-4290-9f2c-94053f8d2458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:42:51.289486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289446    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-model-cache\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:51.289486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289462    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-home\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:51.289583 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.289524    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32c40ca4-c612-4290-9f2c-94053f8d2458" (UID: "32c40ca4-c612-4290-9f2c-94053f8d2458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:42:51.291356 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.291302    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c40ca4-c612-4290-9f2c-94053f8d2458-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32c40ca4-c612-4290-9f2c-94053f8d2458" (UID: "32c40ca4-c612-4290-9f2c-94053f8d2458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:42:51.291469 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.291370    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-dshm" (OuterVolumeSpecName: "dshm") pod "32c40ca4-c612-4290-9f2c-94053f8d2458" (UID: "32c40ca4-c612-4290-9f2c-94053f8d2458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:42:51.291469 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.291395    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c40ca4-c612-4290-9f2c-94053f8d2458-kube-api-access-ggtmf" (OuterVolumeSpecName: "kube-api-access-ggtmf") pod "32c40ca4-c612-4290-9f2c-94053f8d2458" (UID: "32c40ca4-c612-4290-9f2c-94053f8d2458"). InnerVolumeSpecName "kube-api-access-ggtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:42:51.389989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.389966    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtmf\" (UniqueName: \"kubernetes.io/projected/32c40ca4-c612-4290-9f2c-94053f8d2458-kube-api-access-ggtmf\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:51.389989 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.389988    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-tmp-dir\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:51.390110 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.389999    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c40ca4-c612-4290-9f2c-94053f8d2458-dshm\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:51.390110 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.390007    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c40ca4-c612-4290-9f2c-94053f8d2458-tls-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:42:51.658090 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.658058    2577 generic.go:356] "Generic (PLEG): container finished" podID="32c40ca4-c612-4290-9f2c-94053f8d2458" containerID="057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d" exitCode=0
Jul 08 23:42:51.658187 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.658121    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"
Jul 08 23:42:51.658187 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.658147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc" event={"ID":"32c40ca4-c612-4290-9f2c-94053f8d2458","Type":"ContainerDied","Data":"057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d"}
Jul 08 23:42:51.658258 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.658193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc" event={"ID":"32c40ca4-c612-4290-9f2c-94053f8d2458","Type":"ContainerDied","Data":"d85c77971eb622f5d692e8414c0a4d9361db69b5a17b8c4cc2ae042cf1b22f69"}
Jul 08 23:42:51.658258 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.658212    2577 scope.go:117] "RemoveContainer" containerID="057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d"
Jul 08 23:42:51.668358 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.668333    2577 scope.go:117] "RemoveContainer" containerID="057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d"
Jul 08 23:42:51.668814 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:42:51.668791    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d\": container with ID starting with 057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d not found: ID does not exist" containerID="057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d"
Jul 08 23:42:51.668884 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.668826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d"} err="failed to get container status \"057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d\": rpc error: code = NotFound desc = could not find container \"057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d\": container with ID starting with 057a1fc7d497e681f33315ab040a631610d1d509963c3faa979e9d5b8cc7dd7d not found: ID does not exist"
Jul 08 23:42:51.673782 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.673761    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"]
Jul 08 23:42:51.679383 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:51.679361    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-08d361/test-model-08d361-kserve-7dcb84ffd5-mplqc"]
Jul 08 23:42:53.519966 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:42:53.519925    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c40ca4-c612-4290-9f2c-94053f8d2458" path="/var/lib/kubelet/pods/32c40ca4-c612-4290-9f2c-94053f8d2458/volumes"
Jul 08 23:43:20.629685 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.629648    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"]
Jul 08 23:43:20.630208 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.629909    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" podUID="f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" containerName="maas-api" containerID="cri-o://157b8f4c24fb81bbda5cfa94d4783b3f2ab659eac8fde63e453b7f3ca3704a44" gracePeriod=30
Jul 08 23:43:20.792092 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.792059    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" containerID="157b8f4c24fb81bbda5cfa94d4783b3f2ab659eac8fde63e453b7f3ca3704a44" exitCode=0
Jul 08 23:43:20.792226 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.792105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" event={"ID":"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221","Type":"ContainerDied","Data":"157b8f4c24fb81bbda5cfa94d4783b3f2ab659eac8fde63e453b7f3ca3704a44"}
Jul 08 23:43:20.870446 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.870423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:43:20.897643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.897583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-openshift-service-ca\") pod \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") "
Jul 08 23:43:20.897643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.897635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c2z7\" (UniqueName: \"kubernetes.io/projected/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-kube-api-access-5c2z7\") pod \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") "
Jul 08 23:43:20.897829 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.897679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls\") pod \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\" (UID: \"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221\") "
Jul 08 23:43:20.897958 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.897923    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" (UID: "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:43:20.899585 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.899558    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" (UID: "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:43:20.899671 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.899622    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-kube-api-access-5c2z7" (OuterVolumeSpecName: "kube-api-access-5c2z7") pod "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" (UID: "f6048b80-5d4f-44f9-85b5-4ce6bcd1e221"). InnerVolumeSpecName "kube-api-access-5c2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:43:20.998686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.998665    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:20.998686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.998685    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c2z7\" (UniqueName: \"kubernetes.io/projected/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-kube-api-access-5c2z7\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:20.998840 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:20.998694    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:21.797862 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:21.797783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp" event={"ID":"f6048b80-5d4f-44f9-85b5-4ce6bcd1e221","Type":"ContainerDied","Data":"0da5f53071aaa5ce85f47195d1c0a4c7c64014fc41a77e3cc208f5bf8d2d1127"}
Jul 08 23:43:21.797862 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:21.797832    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"
Jul 08 23:43:21.797862 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:21.797857    2577 scope.go:117] "RemoveContainer" containerID="157b8f4c24fb81bbda5cfa94d4783b3f2ab659eac8fde63e453b7f3ca3704a44"
Jul 08 23:43:21.813124 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:21.813095    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"]
Jul 08 23:43:21.817531 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:21.817506    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-717c2b-677c898c77-7j5jp"]
Jul 08 23:43:23.520083 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:23.520053    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" path="/var/lib/kubelet/pods/f6048b80-5d4f-44f9-85b5-4ce6bcd1e221/volumes"
Jul 08 23:43:30.661597 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661565    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"]
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661887    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" containerName="maas-api"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661898    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" containerName="maas-api"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0469a3f1-3f20-4776-b353-b7a55ce83032" containerName="maas-api"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661912    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0469a3f1-3f20-4776-b353-b7a55ce83032" containerName="maas-api"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661922    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c40ca4-c612-4290-9f2c-94053f8d2458" containerName="main"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661928    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c40ca4-c612-4290-9f2c-94053f8d2458" containerName="main"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661987    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0469a3f1-3f20-4776-b353-b7a55ce83032" containerName="maas-api"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.661995    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6048b80-5d4f-44f9-85b5-4ce6bcd1e221" containerName="maas-api"
Jul 08 23:43:30.662065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.662002    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c40ca4-c612-4290-9f2c-94053f8d2458" containerName="main"
Jul 08 23:43:30.666215 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.666193    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.668332 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.668307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-932212-serving-cert\""
Jul 08 23:43:30.676308 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.676284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"]
Jul 08 23:43:30.768383 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.768354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55gkz\" (UniqueName: \"kubernetes.io/projected/53614fad-d885-4508-a67e-6495eb85ab3c-kube-api-access-55gkz\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.768505 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.768418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53614fad-d885-4508-a67e-6495eb85ab3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.768505 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.768480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.869006 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.868973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.869144 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.869032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55gkz\" (UniqueName: \"kubernetes.io/projected/53614fad-d885-4508-a67e-6495eb85ab3c-kube-api-access-55gkz\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.869144 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:30.869119    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-932212-serving-cert: secret "maas-api-e2e-api-a-932212-serving-cert" not found
Jul 08 23:43:30.869144 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.869127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53614fad-d885-4508-a67e-6495eb85ab3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.869302 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:30.869175    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls podName:53614fad-d885-4508-a67e-6495eb85ab3c nodeName:}" failed. No retries permitted until 2026-07-08 23:43:31.369157725 +0000 UTC m=+2160.360991008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls") pod "maas-api-e2e-api-a-932212-865454f8fb-7wq4j" (UID: "53614fad-d885-4508-a67e-6495eb85ab3c") : secret "maas-api-e2e-api-a-932212-serving-cert" not found
Jul 08 23:43:30.869720 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.869685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53614fad-d885-4508-a67e-6495eb85ab3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:30.879429 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:30.879407    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55gkz\" (UniqueName: \"kubernetes.io/projected/53614fad-d885-4508-a67e-6495eb85ab3c-kube-api-access-55gkz\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:31.372569 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:31.372541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:31.372726 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:31.372636    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-932212-serving-cert: secret "maas-api-e2e-api-a-932212-serving-cert" not found
Jul 08 23:43:31.372726 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:31.372681    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls podName:53614fad-d885-4508-a67e-6495eb85ab3c nodeName:}" failed. No retries permitted until 2026-07-08 23:43:32.372666406 +0000 UTC m=+2161.364499690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls") pod "maas-api-e2e-api-a-932212-865454f8fb-7wq4j" (UID: "53614fad-d885-4508-a67e-6495eb85ab3c") : secret "maas-api-e2e-api-a-932212-serving-cert" not found
Jul 08 23:43:32.378137 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:32.378098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:32.380717 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:32.380676    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") pod \"maas-api-e2e-api-a-932212-865454f8fb-7wq4j\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:32.478080 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:32.478056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:32.606798 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:32.606776    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"]
Jul 08 23:43:32.609077 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:43:32.609049    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53614fad_d885_4508_a67e_6495eb85ab3c.slice/crio-74a816a4a3da4e9e11b00dafcd93ff6b7acbb59bbe2566283b2a70350514054f WatchSource:0}: Error finding container 74a816a4a3da4e9e11b00dafcd93ff6b7acbb59bbe2566283b2a70350514054f: Status 404 returned error can't find the container with id 74a816a4a3da4e9e11b00dafcd93ff6b7acbb59bbe2566283b2a70350514054f
Jul 08 23:43:32.844616 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:32.844537    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j" event={"ID":"53614fad-d885-4508-a67e-6495eb85ab3c","Type":"ContainerStarted","Data":"74a816a4a3da4e9e11b00dafcd93ff6b7acbb59bbe2566283b2a70350514054f"}
Jul 08 23:43:33.850807 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:33.850770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j" event={"ID":"53614fad-d885-4508-a67e-6495eb85ab3c","Type":"ContainerStarted","Data":"6dde3ebc202d531282a6a2a6c15f929c60ef5be89adaafd10a2264f6935251dd"}
Jul 08 23:43:33.851208 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:33.850854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:33.867026 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:33.866980    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j" podStartSLOduration=3.447241363 podStartE2EDuration="3.866965455s" totalImagesPullingTime="419.724092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:43:33.866209403 +0000 UTC m=+2162.858042710" watchObservedRunningTime="2026-07-08 23:43:33.866965455 +0000 UTC m=+2162.858798796"
Jul 08 23:43:38.696659 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.696621    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"]
Jul 08 23:43:38.700757 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.700730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.704369 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.704347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1f3801-serving-cert\""
Jul 08 23:43:38.712293 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.712270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"]
Jul 08 23:43:38.825271 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.825239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.825271 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.825271    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc42c\" (UniqueName: \"kubernetes.io/projected/be792469-5b70-450d-aa68-55200503e82f-kube-api-access-pc42c\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.825441 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.825312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be792469-5b70-450d-aa68-55200503e82f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.926269 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.926241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.926436 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.926272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc42c\" (UniqueName: \"kubernetes.io/projected/be792469-5b70-450d-aa68-55200503e82f-kube-api-access-pc42c\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.926436 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.926318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be792469-5b70-450d-aa68-55200503e82f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.926436 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:38.926400    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-serving-cert: secret "maas-api-e2e-api-b-1f3801-serving-cert" not found
Jul 08 23:43:38.926604 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:38.926475    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls podName:be792469-5b70-450d-aa68-55200503e82f nodeName:}" failed. No retries permitted until 2026-07-08 23:43:39.426455274 +0000 UTC m=+2168.418288566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls") pod "maas-api-e2e-api-b-1f3801-75d456d549-dms85" (UID: "be792469-5b70-450d-aa68-55200503e82f") : secret "maas-api-e2e-api-b-1f3801-serving-cert" not found
Jul 08 23:43:38.926898 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.926881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be792469-5b70-450d-aa68-55200503e82f-openshift-service-ca\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:38.935295 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:38.935267    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc42c\" (UniqueName: \"kubernetes.io/projected/be792469-5b70-450d-aa68-55200503e82f-kube-api-access-pc42c\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:39.431540 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:39.431492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:39.433917 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:39.433891    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls\") pod \"maas-api-e2e-api-b-1f3801-75d456d549-dms85\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:39.615891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:39.615857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:39.748405 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:39.748380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"]
Jul 08 23:43:39.750737 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:43:39.750694    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe792469_5b70_450d_aa68_55200503e82f.slice/crio-63194dd54c94da590efe2e92527bbb2189b355948b70a3ce758150f93a3834e5 WatchSource:0}: Error finding container 63194dd54c94da590efe2e92527bbb2189b355948b70a3ce758150f93a3834e5: Status 404 returned error can't find the container with id 63194dd54c94da590efe2e92527bbb2189b355948b70a3ce758150f93a3834e5
Jul 08 23:43:39.861782 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:39.861761    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:39.878644 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:39.878616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85" event={"ID":"be792469-5b70-450d-aa68-55200503e82f","Type":"ContainerStarted","Data":"63194dd54c94da590efe2e92527bbb2189b355948b70a3ce758150f93a3834e5"}
Jul 08 23:43:40.884576 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:40.884544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85" event={"ID":"be792469-5b70-450d-aa68-55200503e82f","Type":"ContainerStarted","Data":"69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f"}
Jul 08 23:43:40.884964 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:40.884630    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:40.901230 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:40.901186    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85" podStartSLOduration=2.557996776 podStartE2EDuration="2.901171323s" totalImagesPullingTime="343.174547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:43:40.900029007 +0000 UTC m=+2169.891862314" watchObservedRunningTime="2026-07-08 23:43:40.901171323 +0000 UTC m=+2169.893004675"
Jul 08 23:43:46.894598 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:46.894565    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:51.807493 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:51.807462    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"]
Jul 08 23:43:51.807998 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:51.807673    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j" podUID="53614fad-d885-4508-a67e-6495eb85ab3c" containerName="maas-api" containerID="cri-o://6dde3ebc202d531282a6a2a6c15f929c60ef5be89adaafd10a2264f6935251dd" gracePeriod=30
Jul 08 23:43:51.936806 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:51.936766    2577 generic.go:356] "Generic (PLEG): container finished" podID="53614fad-d885-4508-a67e-6495eb85ab3c" containerID="6dde3ebc202d531282a6a2a6c15f929c60ef5be89adaafd10a2264f6935251dd" exitCode=0
Jul 08 23:43:51.936922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:51.936808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j" event={"ID":"53614fad-d885-4508-a67e-6495eb85ab3c","Type":"ContainerDied","Data":"6dde3ebc202d531282a6a2a6c15f929c60ef5be89adaafd10a2264f6935251dd"}
Jul 08 23:43:52.047911 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.047888    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:52.124140 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.124117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55gkz\" (UniqueName: \"kubernetes.io/projected/53614fad-d885-4508-a67e-6495eb85ab3c-kube-api-access-55gkz\") pod \"53614fad-d885-4508-a67e-6495eb85ab3c\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") "
Jul 08 23:43:52.124286 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.124160    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53614fad-d885-4508-a67e-6495eb85ab3c-openshift-service-ca\") pod \"53614fad-d885-4508-a67e-6495eb85ab3c\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") "
Jul 08 23:43:52.124286 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.124199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") pod \"53614fad-d885-4508-a67e-6495eb85ab3c\" (UID: \"53614fad-d885-4508-a67e-6495eb85ab3c\") "
Jul 08 23:43:52.124549 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.124529    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53614fad-d885-4508-a67e-6495eb85ab3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53614fad-d885-4508-a67e-6495eb85ab3c" (UID: "53614fad-d885-4508-a67e-6495eb85ab3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:43:52.126135 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.126110    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53614fad-d885-4508-a67e-6495eb85ab3c" (UID: "53614fad-d885-4508-a67e-6495eb85ab3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:43:52.126220 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.126195    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53614fad-d885-4508-a67e-6495eb85ab3c-kube-api-access-55gkz" (OuterVolumeSpecName: "kube-api-access-55gkz") pod "53614fad-d885-4508-a67e-6495eb85ab3c" (UID: "53614fad-d885-4508-a67e-6495eb85ab3c"). InnerVolumeSpecName "kube-api-access-55gkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:43:52.225174 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.225151    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53614fad-d885-4508-a67e-6495eb85ab3c-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:52.225174 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.225172    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53614fad-d885-4508-a67e-6495eb85ab3c-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:52.225289 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.225181    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55gkz\" (UniqueName: \"kubernetes.io/projected/53614fad-d885-4508-a67e-6495eb85ab3c-kube-api-access-55gkz\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:52.943123 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.943093    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"
Jul 08 23:43:52.943536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.943132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j" event={"ID":"53614fad-d885-4508-a67e-6495eb85ab3c","Type":"ContainerDied","Data":"74a816a4a3da4e9e11b00dafcd93ff6b7acbb59bbe2566283b2a70350514054f"}
Jul 08 23:43:52.943536 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.943197    2577 scope.go:117] "RemoveContainer" containerID="6dde3ebc202d531282a6a2a6c15f929c60ef5be89adaafd10a2264f6935251dd"
Jul 08 23:43:52.964543 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.964516    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"]
Jul 08 23:43:52.968431 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:52.968411    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-932212-865454f8fb-7wq4j"]
Jul 08 23:43:53.520049 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:53.520015    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53614fad-d885-4508-a67e-6495eb85ab3c" path="/var/lib/kubelet/pods/53614fad-d885-4508-a67e-6495eb85ab3c/volumes"
Jul 08 23:43:58.629378 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.629348    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"]
Jul 08 23:43:58.629875 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.629554    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85" podUID="be792469-5b70-450d-aa68-55200503e82f" containerName="maas-api" containerID="cri-o://69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f" gracePeriod=30
Jul 08 23:43:58.866913 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.866892    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:58.973327 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.973267    2577 generic.go:356] "Generic (PLEG): container finished" podID="be792469-5b70-450d-aa68-55200503e82f" containerID="69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f" exitCode=0
Jul 08 23:43:58.973327 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.973310    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"
Jul 08 23:43:58.973465 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.973330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85" event={"ID":"be792469-5b70-450d-aa68-55200503e82f","Type":"ContainerDied","Data":"69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f"}
Jul 08 23:43:58.973465 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.973374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85" event={"ID":"be792469-5b70-450d-aa68-55200503e82f","Type":"ContainerDied","Data":"63194dd54c94da590efe2e92527bbb2189b355948b70a3ce758150f93a3834e5"}
Jul 08 23:43:58.973465 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.973397    2577 scope.go:117] "RemoveContainer" containerID="69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f"
Jul 08 23:43:58.978802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.978779    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be792469-5b70-450d-aa68-55200503e82f-openshift-service-ca\") pod \"be792469-5b70-450d-aa68-55200503e82f\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") "
Jul 08 23:43:58.978897 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.978870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls\") pod \"be792469-5b70-450d-aa68-55200503e82f\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") "
Jul 08 23:43:58.978937 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.978908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc42c\" (UniqueName: \"kubernetes.io/projected/be792469-5b70-450d-aa68-55200503e82f-kube-api-access-pc42c\") pod \"be792469-5b70-450d-aa68-55200503e82f\" (UID: \"be792469-5b70-450d-aa68-55200503e82f\") "
Jul 08 23:43:58.979103 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.979082    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be792469-5b70-450d-aa68-55200503e82f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be792469-5b70-450d-aa68-55200503e82f" (UID: "be792469-5b70-450d-aa68-55200503e82f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:43:58.979200 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.979180    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be792469-5b70-450d-aa68-55200503e82f-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:58.980935 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.980911    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be792469-5b70-450d-aa68-55200503e82f-kube-api-access-pc42c" (OuterVolumeSpecName: "kube-api-access-pc42c") pod "be792469-5b70-450d-aa68-55200503e82f" (UID: "be792469-5b70-450d-aa68-55200503e82f"). InnerVolumeSpecName "kube-api-access-pc42c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:43:58.981193 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.981167    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be792469-5b70-450d-aa68-55200503e82f" (UID: "be792469-5b70-450d-aa68-55200503e82f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:43:58.982349 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.982328    2577 scope.go:117] "RemoveContainer" containerID="69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f"
Jul 08 23:43:58.982609 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:43:58.982589    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f\": container with ID starting with 69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f not found: ID does not exist" containerID="69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f"
Jul 08 23:43:58.982667 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:58.982620    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f"} err="failed to get container status \"69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f\": rpc error: code = NotFound desc = could not find container \"69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f\": container with ID starting with 69846699f3951a24e858131c1bd33a863ffd213e0c22a2e513133612c431f67f not found: ID does not exist"
Jul 08 23:43:59.080136 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:59.080100    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be792469-5b70-450d-aa68-55200503e82f-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:59.080136 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:59.080136    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc42c\" (UniqueName: \"kubernetes.io/projected/be792469-5b70-450d-aa68-55200503e82f-kube-api-access-pc42c\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:43:59.296564 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:59.296541    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"]
Jul 08 23:43:59.307411 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:59.307388    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1f3801-75d456d549-dms85"]
Jul 08 23:43:59.519493 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:43:59.519464    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be792469-5b70-450d-aa68-55200503e82f" path="/var/lib/kubelet/pods/be792469-5b70-450d-aa68-55200503e82f/volumes"
Jul 08 23:44:07.915663 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.915630    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"]
Jul 08 23:44:07.916052 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.915979    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53614fad-d885-4508-a67e-6495eb85ab3c" containerName="maas-api"
Jul 08 23:44:07.916052 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.915991    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="53614fad-d885-4508-a67e-6495eb85ab3c" containerName="maas-api"
Jul 08 23:44:07.916052 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.916010    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be792469-5b70-450d-aa68-55200503e82f" containerName="maas-api"
Jul 08 23:44:07.916052 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.916015    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be792469-5b70-450d-aa68-55200503e82f" containerName="maas-api"
Jul 08 23:44:07.916205 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.916069    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="53614fad-d885-4508-a67e-6495eb85ab3c" containerName="maas-api"
Jul 08 23:44:07.916205 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.916078    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be792469-5b70-450d-aa68-55200503e82f" containerName="maas-api"
Jul 08 23:44:07.918891 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.918867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:07.921105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.921081    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-310c89-serving-cert\""
Jul 08 23:44:07.928531 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:07.928506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"]
Jul 08 23:44:08.052301 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.052266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.052301 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.052308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02471e12-5645-4d14-9e7e-56978b2a8f66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.052476 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.052420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6cs\" (UniqueName: \"kubernetes.io/projected/02471e12-5645-4d14-9e7e-56978b2a8f66-kube-api-access-zb6cs\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.153007 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.152979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6cs\" (UniqueName: \"kubernetes.io/projected/02471e12-5645-4d14-9e7e-56978b2a8f66-kube-api-access-zb6cs\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.153126 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.153013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.153126 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.153034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02471e12-5645-4d14-9e7e-56978b2a8f66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.153206 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:44:08.153125    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-serving-cert: secret "maas-api-e2e-shared-a-310c89-serving-cert" not found
Jul 08 23:44:08.153206 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:44:08.153191    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls podName:02471e12-5645-4d14-9e7e-56978b2a8f66 nodeName:}" failed. No retries permitted until 2026-07-08 23:44:08.653174492 +0000 UTC m=+2197.645007776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls") pod "maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" (UID: "02471e12-5645-4d14-9e7e-56978b2a8f66") : secret "maas-api-e2e-shared-a-310c89-serving-cert" not found
Jul 08 23:44:08.153593 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.153577    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02471e12-5645-4d14-9e7e-56978b2a8f66-openshift-service-ca\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.162186 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.162162    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6cs\" (UniqueName: \"kubernetes.io/projected/02471e12-5645-4d14-9e7e-56978b2a8f66-kube-api-access-zb6cs\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.657482 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.657447    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.659718 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.659685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls\") pod \"maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.830202 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.830176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:08.954785 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:08.954761    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"]
Jul 08 23:44:08.957170 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:44:08.957138    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02471e12_5645_4d14_9e7e_56978b2a8f66.slice/crio-a6a426bf800ded70be75674e2f939b783113a34168878d6fa0ded1721728acf4 WatchSource:0}: Error finding container a6a426bf800ded70be75674e2f939b783113a34168878d6fa0ded1721728acf4: Status 404 returned error can't find the container with id a6a426bf800ded70be75674e2f939b783113a34168878d6fa0ded1721728acf4
Jul 08 23:44:09.020266 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:09.020236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" event={"ID":"02471e12-5645-4d14-9e7e-56978b2a8f66","Type":"ContainerStarted","Data":"a6a426bf800ded70be75674e2f939b783113a34168878d6fa0ded1721728acf4"}
Jul 08 23:44:10.025818 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:10.025777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" event={"ID":"02471e12-5645-4d14-9e7e-56978b2a8f66","Type":"ContainerStarted","Data":"e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab"}
Jul 08 23:44:10.026229 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:10.025849    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:10.042642 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:10.042592    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" podStartSLOduration=2.672673165 podStartE2EDuration="3.042578221s" totalImagesPullingTime="369.905056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:44:10.042080043 +0000 UTC m=+2199.033913349" watchObservedRunningTime="2026-07-08 23:44:10.042578221 +0000 UTC m=+2199.034411527"
Jul 08 23:44:16.037792 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.037764    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:44:16.108598 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.108568    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"]
Jul 08 23:44:16.112394 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.112375    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.114963 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.114930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-cd1865-serving-cert\""
Jul 08 23:44:16.122888 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.122865    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"]
Jul 08 23:44:16.217824 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.217797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kkq\" (UniqueName: \"kubernetes.io/projected/97ac65b5-c4b9-4916-9a7f-64c692bf249a-kube-api-access-b6kkq\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.218032 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.218006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac65b5-c4b9-4916-9a7f-64c692bf249a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.218100 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.218071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97ac65b5-c4b9-4916-9a7f-64c692bf249a-maas-api-tls\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.319491 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.319434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kkq\" (UniqueName: \"kubernetes.io/projected/97ac65b5-c4b9-4916-9a7f-64c692bf249a-kube-api-access-b6kkq\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.319491 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.319473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac65b5-c4b9-4916-9a7f-64c692bf249a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.319621 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.319506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97ac65b5-c4b9-4916-9a7f-64c692bf249a-maas-api-tls\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.320074 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.320044    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac65b5-c4b9-4916-9a7f-64c692bf249a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.321886 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.321865    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97ac65b5-c4b9-4916-9a7f-64c692bf249a-maas-api-tls\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.338205 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.338182    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kkq\" (UniqueName: \"kubernetes.io/projected/97ac65b5-c4b9-4916-9a7f-64c692bf249a-kube-api-access-b6kkq\") pod \"maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.423858 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.423832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:16.553572 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:16.553549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"]
Jul 08 23:44:16.555503 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:44:16.555475    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ac65b5_c4b9_4916_9a7f_64c692bf249a.slice/crio-bd0de1b65046749111f8f68f2e70fe6e7930e79edc0921a5e4677ce4467ef6f1 WatchSource:0}: Error finding container bd0de1b65046749111f8f68f2e70fe6e7930e79edc0921a5e4677ce4467ef6f1: Status 404 returned error can't find the container with id bd0de1b65046749111f8f68f2e70fe6e7930e79edc0921a5e4677ce4467ef6f1
Jul 08 23:44:17.059787 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:17.059741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw" event={"ID":"97ac65b5-c4b9-4916-9a7f-64c692bf249a","Type":"ContainerStarted","Data":"4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b"}
Jul 08 23:44:17.060131 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:17.059811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw" event={"ID":"97ac65b5-c4b9-4916-9a7f-64c692bf249a","Type":"ContainerStarted","Data":"bd0de1b65046749111f8f68f2e70fe6e7930e79edc0921a5e4677ce4467ef6f1"}
Jul 08 23:44:17.060131 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:17.059843    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:17.081543 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:17.081497    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw" podStartSLOduration=0.754833806 podStartE2EDuration="1.08148304s" totalImagesPullingTime="326.649234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:44:17.080091555 +0000 UTC m=+2206.071924875" watchObservedRunningTime="2026-07-08 23:44:17.08148304 +0000 UTC m=+2206.073316349"
Jul 08 23:44:23.069849 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:23.069814    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:44:31.721213 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.721183    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"]
Jul 08 23:44:31.726043 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.726024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.729429 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.729409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"default-dockercfg-ccd7r\""
Jul 08 23:44:31.729626 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.729612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"openshift-service-ca.crt\""
Jul 08 23:44:31.729681 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.729634    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"kube-root-ca.crt\""
Jul 08 23:44:31.729783 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.729763    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"auth-test-model-310c89-kserve-self-signed-certs\""
Jul 08 23:44:31.733534 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.733508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0537ff41-8113-4ca0-8920-e51ba92cf566-tls-certs\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.733749 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.733724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-dshm\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.733876 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.733781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-model-cache\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.733876 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.733815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-tmp-dir\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.733876 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.733841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5vc\" (UniqueName: \"kubernetes.io/projected/0537ff41-8113-4ca0-8920-e51ba92cf566-kube-api-access-vb5vc\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.734046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.733876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-home\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.735461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.735439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"]
Jul 08 23:44:31.834188 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0537ff41-8113-4ca0-8920-e51ba92cf566-tls-certs\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834283 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-dshm\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834283 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-model-cache\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834283 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-tmp-dir\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834283 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5vc\" (UniqueName: \"kubernetes.io/projected/0537ff41-8113-4ca0-8920-e51ba92cf566-kube-api-access-vb5vc\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834460 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-home\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834664 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834633    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-model-cache\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834671    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-tmp-dir\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.834768 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.834697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-home\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.836374 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.836358    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-dshm\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.836657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.836639    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0537ff41-8113-4ca0-8920-e51ba92cf566-tls-certs\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:31.842001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:31.841980    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5vc\" (UniqueName: \"kubernetes.io/projected/0537ff41-8113-4ca0-8920-e51ba92cf566-kube-api-access-vb5vc\") pod \"auth-test-model-310c89-kserve-6cd7c674cf-l5w6m\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") " pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:32.036148 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:32.036124    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:32.169032 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:32.169010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"]
Jul 08 23:44:32.171479 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:44:32.171448    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0537ff41_8113_4ca0_8920_e51ba92cf566.slice/crio-3010b4510b333509d96fee101968c8d4a93ff2e815cd6e0249a7bcd5e150f6c8 WatchSource:0}: Error finding container 3010b4510b333509d96fee101968c8d4a93ff2e815cd6e0249a7bcd5e150f6c8: Status 404 returned error can't find the container with id 3010b4510b333509d96fee101968c8d4a93ff2e815cd6e0249a7bcd5e150f6c8
Jul 08 23:44:33.137925 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:33.137882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m" event={"ID":"0537ff41-8113-4ca0-8920-e51ba92cf566","Type":"ContainerStarted","Data":"32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb"}
Jul 08 23:44:33.138324 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:33.137941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m" event={"ID":"0537ff41-8113-4ca0-8920-e51ba92cf566","Type":"ContainerStarted","Data":"3010b4510b333509d96fee101968c8d4a93ff2e815cd6e0249a7bcd5e150f6c8"}
Jul 08 23:44:33.155381 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:33.155330    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m" podStartSLOduration=2.1553149 podStartE2EDuration="2.1553149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:44:33.153620406 +0000 UTC m=+2222.145453716" watchObservedRunningTime="2026-07-08 23:44:33.1553149 +0000 UTC m=+2222.147148205"
Jul 08 23:44:42.036875 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:42.036842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:42.036875 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:42.036884    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:42.049361 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:42.049337    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:42.192523 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:42.192494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:44:52.901957 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.901920    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"]
Jul 08 23:44:52.905503 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.905487    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:52.907982 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.907962    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"openshift-service-ca.crt\""
Jul 08 23:44:52.909056 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.909033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"default-dockercfg-zlxb5\""
Jul 08 23:44:52.909157 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.909047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"kube-root-ca.crt\""
Jul 08 23:44:52.909157 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.909115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"auth-test-model-cd1865-kserve-self-signed-certs\""
Jul 08 23:44:52.913324 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.913295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"]
Jul 08 23:44:52.988642 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.988616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-dshm\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:52.988767 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.988651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jx54\" (UniqueName: \"kubernetes.io/projected/09d8e7bf-c38b-4abc-b318-f648ecd45097-kube-api-access-5jx54\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:52.988767 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.988690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-model-cache\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:52.988854 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.988766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-home\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:52.988854 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.988807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d8e7bf-c38b-4abc-b318-f648ecd45097-tls-certs\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:52.988854 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:52.988833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-tmp-dir\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.089829 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.089759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jx54\" (UniqueName: \"kubernetes.io/projected/09d8e7bf-c38b-4abc-b318-f648ecd45097-kube-api-access-5jx54\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.089952 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.089904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-model-cache\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.089952 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.089932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-home\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.090030 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.089995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d8e7bf-c38b-4abc-b318-f648ecd45097-tls-certs\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.090030 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.090023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-tmp-dir\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.090107 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.090054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-dshm\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.090344 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.090316    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-model-cache\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.090470 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.090363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-home\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.090470 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.090388    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-tmp-dir\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.092151 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.092129    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-dshm\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.092392 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.092376    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d8e7bf-c38b-4abc-b318-f648ecd45097-tls-certs\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.104846 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.104819    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jx54\" (UniqueName: \"kubernetes.io/projected/09d8e7bf-c38b-4abc-b318-f648ecd45097-kube-api-access-5jx54\") pod \"auth-test-model-cd1865-kserve-58ccc6c68c-vldx8\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") " pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.215764 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.215694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:44:53.342572 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:53.342548    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"]
Jul 08 23:44:53.344528 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:44:53.344496    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d8e7bf_c38b_4abc_b318_f648ecd45097.slice/crio-c2730b8eb100f24fdbcdca93786ffc4962ca0e8f4de077bf80299a0b38a42140 WatchSource:0}: Error finding container c2730b8eb100f24fdbcdca93786ffc4962ca0e8f4de077bf80299a0b38a42140: Status 404 returned error can't find the container with id c2730b8eb100f24fdbcdca93786ffc4962ca0e8f4de077bf80299a0b38a42140
Jul 08 23:44:54.238838 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:54.238801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8" event={"ID":"09d8e7bf-c38b-4abc-b318-f648ecd45097","Type":"ContainerStarted","Data":"ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b"}
Jul 08 23:44:54.239218 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:54.238851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8" event={"ID":"09d8e7bf-c38b-4abc-b318-f648ecd45097","Type":"ContainerStarted","Data":"c2730b8eb100f24fdbcdca93786ffc4962ca0e8f4de077bf80299a0b38a42140"}
Jul 08 23:44:54.254889 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:44:54.254839    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8" podStartSLOduration=2.254822885 podStartE2EDuration="2.254822885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:44:54.253600699 +0000 UTC m=+2243.245434005" watchObservedRunningTime="2026-07-08 23:44:54.254822885 +0000 UTC m=+2243.246656191"
Jul 08 23:45:03.216786 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:45:03.216751    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:45:03.216786 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:45:03.216794    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:45:03.228861 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:45:03.228833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:45:03.295142 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:45:03.295113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:47:09.643046 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:09.643016    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"]
Jul 08 23:47:10.518574 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:10.518546    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"]
Jul 08 23:47:24.664410 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.664361    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m" podUID="0537ff41-8113-4ca0-8920-e51ba92cf566" containerName="main" containerID="cri-o://32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb" gracePeriod=45
Jul 08 23:47:24.922491 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.922441    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:47:24.930100 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.930074    2577 generic.go:356] "Generic (PLEG): container finished" podID="0537ff41-8113-4ca0-8920-e51ba92cf566" containerID="32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb" exitCode=0
Jul 08 23:47:24.930198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.930108    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"
Jul 08 23:47:24.930198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.930121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m" event={"ID":"0537ff41-8113-4ca0-8920-e51ba92cf566","Type":"ContainerDied","Data":"32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb"}
Jul 08 23:47:24.930198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.930159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m" event={"ID":"0537ff41-8113-4ca0-8920-e51ba92cf566","Type":"ContainerDied","Data":"3010b4510b333509d96fee101968c8d4a93ff2e815cd6e0249a7bcd5e150f6c8"}
Jul 08 23:47:24.930198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.930180    2577 scope.go:117] "RemoveContainer" containerID="32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb"
Jul 08 23:47:24.938119 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.938102    2577 scope.go:117] "RemoveContainer" containerID="32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb"
Jul 08 23:47:24.938402 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:47:24.938382    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb\": container with ID starting with 32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb not found: ID does not exist" containerID="32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb"
Jul 08 23:47:24.938465 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.938411    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb"} err="failed to get container status \"32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb\": rpc error: code = NotFound desc = could not find container \"32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb\": container with ID starting with 32c06c11cc94b3d480606d03698cfdb1df9f34b13c02bb7e0709f4fe985929bb not found: ID does not exist"
Jul 08 23:47:24.939637 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.939622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0537ff41-8113-4ca0-8920-e51ba92cf566-tls-certs\") pod \"0537ff41-8113-4ca0-8920-e51ba92cf566\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") "
Jul 08 23:47:24.939687 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.939665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-tmp-dir\") pod \"0537ff41-8113-4ca0-8920-e51ba92cf566\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") "
Jul 08 23:47:24.939687 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.939681    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-home\") pod \"0537ff41-8113-4ca0-8920-e51ba92cf566\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") "
Jul 08 23:47:24.939786 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.939750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb5vc\" (UniqueName: \"kubernetes.io/projected/0537ff41-8113-4ca0-8920-e51ba92cf566-kube-api-access-vb5vc\") pod \"0537ff41-8113-4ca0-8920-e51ba92cf566\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") "
Jul 08 23:47:24.939837 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.939786    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-dshm\") pod \"0537ff41-8113-4ca0-8920-e51ba92cf566\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") "
Jul 08 23:47:24.939837 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.939825    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-model-cache\") pod \"0537ff41-8113-4ca0-8920-e51ba92cf566\" (UID: \"0537ff41-8113-4ca0-8920-e51ba92cf566\") "
Jul 08 23:47:24.940054 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.940023    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-home" (OuterVolumeSpecName: "home") pod "0537ff41-8113-4ca0-8920-e51ba92cf566" (UID: "0537ff41-8113-4ca0-8920-e51ba92cf566"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:24.940180 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.940120    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-home\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:24.940261 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.940237    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-model-cache" (OuterVolumeSpecName: "model-cache") pod "0537ff41-8113-4ca0-8920-e51ba92cf566" (UID: "0537ff41-8113-4ca0-8920-e51ba92cf566"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:24.940337 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.940242    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0537ff41-8113-4ca0-8920-e51ba92cf566" (UID: "0537ff41-8113-4ca0-8920-e51ba92cf566"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:24.942131 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.942108    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0537ff41-8113-4ca0-8920-e51ba92cf566-kube-api-access-vb5vc" (OuterVolumeSpecName: "kube-api-access-vb5vc") pod "0537ff41-8113-4ca0-8920-e51ba92cf566" (UID: "0537ff41-8113-4ca0-8920-e51ba92cf566"). InnerVolumeSpecName "kube-api-access-vb5vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:47:24.942350 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.942320    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0537ff41-8113-4ca0-8920-e51ba92cf566-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0537ff41-8113-4ca0-8920-e51ba92cf566" (UID: "0537ff41-8113-4ca0-8920-e51ba92cf566"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:47:24.942550 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:24.942529    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-dshm" (OuterVolumeSpecName: "dshm") pod "0537ff41-8113-4ca0-8920-e51ba92cf566" (UID: "0537ff41-8113-4ca0-8920-e51ba92cf566"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:25.040530 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.040509    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-tmp-dir\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.040530 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.040529    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5vc\" (UniqueName: \"kubernetes.io/projected/0537ff41-8113-4ca0-8920-e51ba92cf566-kube-api-access-vb5vc\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.040648 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.040538    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-dshm\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.040648 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.040547    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0537ff41-8113-4ca0-8920-e51ba92cf566-model-cache\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.040648 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.040555    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0537ff41-8113-4ca0-8920-e51ba92cf566-tls-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.251293 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.251268    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"]
Jul 08 23:47:25.254794 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.254766    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/auth-test-model-310c89-kserve-6cd7c674cf-l5w6m"]
Jul 08 23:47:25.520412 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.520352    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0537ff41-8113-4ca0-8920-e51ba92cf566" path="/var/lib/kubelet/pods/0537ff41-8113-4ca0-8920-e51ba92cf566/volumes"
Jul 08 23:47:25.538214 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.538191    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8" podUID="09d8e7bf-c38b-4abc-b318-f648ecd45097" containerName="main" containerID="cri-o://ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b" gracePeriod=45
Jul 08 23:47:25.787080 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.787053    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:47:25.844095 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-home\") pod \"09d8e7bf-c38b-4abc-b318-f648ecd45097\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") "
Jul 08 23:47:25.844203 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-tmp-dir\") pod \"09d8e7bf-c38b-4abc-b318-f648ecd45097\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") "
Jul 08 23:47:25.844203 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844166    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d8e7bf-c38b-4abc-b318-f648ecd45097-tls-certs\") pod \"09d8e7bf-c38b-4abc-b318-f648ecd45097\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") "
Jul 08 23:47:25.844203 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-dshm\") pod \"09d8e7bf-c38b-4abc-b318-f648ecd45097\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") "
Jul 08 23:47:25.844316 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-model-cache\") pod \"09d8e7bf-c38b-4abc-b318-f648ecd45097\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") "
Jul 08 23:47:25.844316 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844274    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jx54\" (UniqueName: \"kubernetes.io/projected/09d8e7bf-c38b-4abc-b318-f648ecd45097-kube-api-access-5jx54\") pod \"09d8e7bf-c38b-4abc-b318-f648ecd45097\" (UID: \"09d8e7bf-c38b-4abc-b318-f648ecd45097\") "
Jul 08 23:47:25.844418 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844327    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-home" (OuterVolumeSpecName: "home") pod "09d8e7bf-c38b-4abc-b318-f648ecd45097" (UID: "09d8e7bf-c38b-4abc-b318-f648ecd45097"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:25.844418 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844408    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09d8e7bf-c38b-4abc-b318-f648ecd45097" (UID: "09d8e7bf-c38b-4abc-b318-f648ecd45097"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:25.844554 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844531    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-home\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.844656 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844555    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-tmp-dir\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.844656 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.844528    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-model-cache" (OuterVolumeSpecName: "model-cache") pod "09d8e7bf-c38b-4abc-b318-f648ecd45097" (UID: "09d8e7bf-c38b-4abc-b318-f648ecd45097"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:25.846377 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.846350    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d8e7bf-c38b-4abc-b318-f648ecd45097-kube-api-access-5jx54" (OuterVolumeSpecName: "kube-api-access-5jx54") pod "09d8e7bf-c38b-4abc-b318-f648ecd45097" (UID: "09d8e7bf-c38b-4abc-b318-f648ecd45097"). InnerVolumeSpecName "kube-api-access-5jx54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:47:25.846462 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.846380    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d8e7bf-c38b-4abc-b318-f648ecd45097-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09d8e7bf-c38b-4abc-b318-f648ecd45097" (UID: "09d8e7bf-c38b-4abc-b318-f648ecd45097"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:47:25.846462 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.846383    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-dshm" (OuterVolumeSpecName: "dshm") pod "09d8e7bf-c38b-4abc-b318-f648ecd45097" (UID: "09d8e7bf-c38b-4abc-b318-f648ecd45097"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:47:25.935461 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.935435    2577 generic.go:356] "Generic (PLEG): container finished" podID="09d8e7bf-c38b-4abc-b318-f648ecd45097" containerID="ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b" exitCode=0
Jul 08 23:47:25.935560 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.935482    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"
Jul 08 23:47:25.935560 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.935531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8" event={"ID":"09d8e7bf-c38b-4abc-b318-f648ecd45097","Type":"ContainerDied","Data":"ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b"}
Jul 08 23:47:25.935672 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.935586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8" event={"ID":"09d8e7bf-c38b-4abc-b318-f648ecd45097","Type":"ContainerDied","Data":"c2730b8eb100f24fdbcdca93786ffc4962ca0e8f4de077bf80299a0b38a42140"}
Jul 08 23:47:25.935672 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.935613    2577 scope.go:117] "RemoveContainer" containerID="ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b"
Jul 08 23:47:25.946498 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.946463    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d8e7bf-c38b-4abc-b318-f648ecd45097-tls-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.946498 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.946496    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-dshm\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.946643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.946511    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d8e7bf-c38b-4abc-b318-f648ecd45097-model-cache\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.946643 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.946526    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jx54\" (UniqueName: \"kubernetes.io/projected/09d8e7bf-c38b-4abc-b318-f648ecd45097-kube-api-access-5jx54\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:47:25.950271 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.950254    2577 scope.go:117] "RemoveContainer" containerID="ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b"
Jul 08 23:47:25.950531 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:47:25.950507    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b\": container with ID starting with ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b not found: ID does not exist" containerID="ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b"
Jul 08 23:47:25.950597 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.950542    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b"} err="failed to get container status \"ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b\": rpc error: code = NotFound desc = could not find container \"ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b\": container with ID starting with ae87d67d400bc82c32d8d50ee4e845e8d5360430fdfc99f1c7ddfb913f7de60b not found: ID does not exist"
Jul 08 23:47:25.959430 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.959410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"]
Jul 08 23:47:25.965590 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:25.965571    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/auth-test-model-cd1865-kserve-58ccc6c68c-vldx8"]
Jul 08 23:47:27.520066 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:27.520028    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d8e7bf-c38b-4abc-b318-f648ecd45097" path="/var/lib/kubelet/pods/09d8e7bf-c38b-4abc-b318-f648ecd45097/volumes"
Jul 08 23:47:50.305143 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305106    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"]
Jul 08 23:47:50.305668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305507    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0537ff41-8113-4ca0-8920-e51ba92cf566" containerName="main"
Jul 08 23:47:50.305668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305526    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0537ff41-8113-4ca0-8920-e51ba92cf566" containerName="main"
Jul 08 23:47:50.305668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d8e7bf-c38b-4abc-b318-f648ecd45097" containerName="main"
Jul 08 23:47:50.305668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305550    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d8e7bf-c38b-4abc-b318-f648ecd45097" containerName="main"
Jul 08 23:47:50.305668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305658    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0537ff41-8113-4ca0-8920-e51ba92cf566" containerName="main"
Jul 08 23:47:50.305668 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.305672    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d8e7bf-c38b-4abc-b318-f648ecd45097" containerName="main"
Jul 08 23:47:50.312727 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.312693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.316613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.316582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"default-dockercfg-zlxb5\""
Jul 08 23:47:50.316613 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.316603    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"kube-root-ca.crt\""
Jul 08 23:47:50.316865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.316585    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"openshift-service-ca.crt\""
Jul 08 23:47:50.316865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.316665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"]
Jul 08 23:47:50.316865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.316779    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cd1865\"/\"sub-test-model-cd1865-kserve-self-signed-certs\""
Jul 08 23:47:50.429842 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.429821    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-home\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.429961 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.429854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-dshm\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.429961 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.429878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjldh\" (UniqueName: \"kubernetes.io/projected/c87d9701-09ae-4045-b67d-ffcea135e61d-kube-api-access-hjldh\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.429961 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.429933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-tmp-dir\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.430304 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.430019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87d9701-09ae-4045-b67d-ffcea135e61d-tls-certs\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.430304 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.430064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-model-cache\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.530752 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.530721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-home\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.530879 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.530764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-dshm\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.530879 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.530787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjldh\" (UniqueName: \"kubernetes.io/projected/c87d9701-09ae-4045-b67d-ffcea135e61d-kube-api-access-hjldh\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.530879 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.530808    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-tmp-dir\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.531054 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.530937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87d9701-09ae-4045-b67d-ffcea135e61d-tls-certs\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.531054 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.530985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-model-cache\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.531139 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.531054    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-home\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.531139 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.531087    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-tmp-dir\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.531293 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.531269    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-model-cache\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.533006 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.532986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-dshm\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.533319 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.533303    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87d9701-09ae-4045-b67d-ffcea135e61d-tls-certs\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.547897 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.547877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjldh\" (UniqueName: \"kubernetes.io/projected/c87d9701-09ae-4045-b67d-ffcea135e61d-kube-api-access-hjldh\") pod \"sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") " pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.622987 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.622958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:47:50.750389 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:50.750365    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"]
Jul 08 23:47:50.752245 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:47:50.752218    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc87d9701_09ae_4045_b67d_ffcea135e61d.slice/crio-5e1bef7d1766d64b56cb164b6babe3c96b5c270a336e474c2f3b0298ab7f8f59 WatchSource:0}: Error finding container 5e1bef7d1766d64b56cb164b6babe3c96b5c270a336e474c2f3b0298ab7f8f59: Status 404 returned error can't find the container with id 5e1bef7d1766d64b56cb164b6babe3c96b5c270a336e474c2f3b0298ab7f8f59
Jul 08 23:47:51.057112 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:51.057014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr" event={"ID":"c87d9701-09ae-4045-b67d-ffcea135e61d","Type":"ContainerStarted","Data":"6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13"}
Jul 08 23:47:51.057112 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:51.057078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr" event={"ID":"c87d9701-09ae-4045-b67d-ffcea135e61d","Type":"ContainerStarted","Data":"5e1bef7d1766d64b56cb164b6babe3c96b5c270a336e474c2f3b0298ab7f8f59"}
Jul 08 23:47:51.073281 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:47:51.073233    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr" podStartSLOduration=1.073218426 podStartE2EDuration="1.073218426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:47:51.072321575 +0000 UTC m=+2420.064154881" watchObservedRunningTime="2026-07-08 23:47:51.073218426 +0000 UTC m=+2420.065051734"
Jul 08 23:48:00.623343 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:00.623303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:48:00.623800 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:00.623381    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:48:00.636086 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:00.636065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:48:01.114164 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:01.114141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:48:27.308079 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:27.308046    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"]
Jul 08 23:48:31.691131 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.691100    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"]
Jul 08 23:48:31.694581 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.694563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.697307 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.697283    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"openshift-service-ca.crt\""
Jul 08 23:48:31.698196 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.698176    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"kube-root-ca.crt\""
Jul 08 23:48:31.698196 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.698188    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"rate-test-model-310c89-kserve-self-signed-certs\""
Jul 08 23:48:31.698352 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.698319    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-310c89\"/\"default-dockercfg-ccd7r\""
Jul 08 23:48:31.705021 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.705002    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"]
Jul 08 23:48:31.851044 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.851021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-model-cache\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.851150 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.851063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-dshm\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.851150 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.851081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85dfh\" (UniqueName: \"kubernetes.io/projected/d9fbbbde-277f-4a0e-b61c-bc47bf351869-kube-api-access-85dfh\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.851150 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.851107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tmp-dir\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.851150 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.851126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tls-certs\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.851150 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.851150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-home\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952385 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-home\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952385 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-model-cache\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952547 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-dshm\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952547 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85dfh\" (UniqueName: \"kubernetes.io/projected/d9fbbbde-277f-4a0e-b61c-bc47bf351869-kube-api-access-85dfh\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952547 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tmp-dir\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952727 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tls-certs\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952751    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-home\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952889 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952867    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-model-cache\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.952941 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.952909    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tmp-dir\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.954635 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.954615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-dshm\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.954983 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.954967    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tls-certs\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:31.962806 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:31.962758    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85dfh\" (UniqueName: \"kubernetes.io/projected/d9fbbbde-277f-4a0e-b61c-bc47bf351869-kube-api-access-85dfh\") pod \"rate-test-model-310c89-kserve-84d6588c5f-h46gl\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") " pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:32.003861 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:32.003831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:32.130029 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:32.130005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"]
Jul 08 23:48:32.131960 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:48:32.131930    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9fbbbde_277f_4a0e_b61c_bc47bf351869.slice/crio-a372fc55839aa18e83a935a0eec244a59d20b46bc2fa8055975a1de712b4e28f WatchSource:0}: Error finding container a372fc55839aa18e83a935a0eec244a59d20b46bc2fa8055975a1de712b4e28f: Status 404 returned error can't find the container with id a372fc55839aa18e83a935a0eec244a59d20b46bc2fa8055975a1de712b4e28f
Jul 08 23:48:32.253669 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:32.253593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl" event={"ID":"d9fbbbde-277f-4a0e-b61c-bc47bf351869","Type":"ContainerStarted","Data":"6cf6d877689fd328054c6f8738f5dedf37a202a6c91108d99bcd122cdf14b073"}
Jul 08 23:48:32.253669 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:32.253638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl" event={"ID":"d9fbbbde-277f-4a0e-b61c-bc47bf351869","Type":"ContainerStarted","Data":"a372fc55839aa18e83a935a0eec244a59d20b46bc2fa8055975a1de712b4e28f"}
Jul 08 23:48:32.272338 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:32.272293    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl" podStartSLOduration=1.272277861 podStartE2EDuration="1.272277861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:48:32.270677269 +0000 UTC m=+2461.262510575" watchObservedRunningTime="2026-07-08 23:48:32.272277861 +0000 UTC m=+2461.264111167"
Jul 08 23:48:42.004300 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.004265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:42.004670 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.004311    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:42.016761 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.016733    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:42.316321 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.316243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:48:42.329443 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.329407    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr" podUID="c87d9701-09ae-4045-b67d-ffcea135e61d" containerName="main" containerID="cri-o://6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13" gracePeriod=45
Jul 08 23:48:42.593557 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.593537    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:48:42.742997 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.742973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-home\") pod \"c87d9701-09ae-4045-b67d-ffcea135e61d\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") "
Jul 08 23:48:42.743105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-tmp-dir\") pod \"c87d9701-09ae-4045-b67d-ffcea135e61d\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") "
Jul 08 23:48:42.743105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-model-cache\") pod \"c87d9701-09ae-4045-b67d-ffcea135e61d\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") "
Jul 08 23:48:42.743105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-dshm\") pod \"c87d9701-09ae-4045-b67d-ffcea135e61d\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") "
Jul 08 23:48:42.743281 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjldh\" (UniqueName: \"kubernetes.io/projected/c87d9701-09ae-4045-b67d-ffcea135e61d-kube-api-access-hjldh\") pod \"c87d9701-09ae-4045-b67d-ffcea135e61d\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") "
Jul 08 23:48:42.743281 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87d9701-09ae-4045-b67d-ffcea135e61d-tls-certs\") pod \"c87d9701-09ae-4045-b67d-ffcea135e61d\" (UID: \"c87d9701-09ae-4045-b67d-ffcea135e61d\") "
Jul 08 23:48:42.743390 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743296    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-home" (OuterVolumeSpecName: "home") pod "c87d9701-09ae-4045-b67d-ffcea135e61d" (UID: "c87d9701-09ae-4045-b67d-ffcea135e61d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:48:42.743390 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743312    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c87d9701-09ae-4045-b67d-ffcea135e61d" (UID: "c87d9701-09ae-4045-b67d-ffcea135e61d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:48:42.743390 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743327    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c87d9701-09ae-4045-b67d-ffcea135e61d" (UID: "c87d9701-09ae-4045-b67d-ffcea135e61d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:48:42.743530 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743478    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-home\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:48:42.743530 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743496    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-tmp-dir\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:48:42.743530 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.743510    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-model-cache\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:48:42.745173 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.745148    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-dshm" (OuterVolumeSpecName: "dshm") pod "c87d9701-09ae-4045-b67d-ffcea135e61d" (UID: "c87d9701-09ae-4045-b67d-ffcea135e61d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:48:42.745376 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.745355    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c87d9701-09ae-4045-b67d-ffcea135e61d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c87d9701-09ae-4045-b67d-ffcea135e61d" (UID: "c87d9701-09ae-4045-b67d-ffcea135e61d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:48:42.745477 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.745380    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87d9701-09ae-4045-b67d-ffcea135e61d-kube-api-access-hjldh" (OuterVolumeSpecName: "kube-api-access-hjldh") pod "c87d9701-09ae-4045-b67d-ffcea135e61d" (UID: "c87d9701-09ae-4045-b67d-ffcea135e61d"). InnerVolumeSpecName "kube-api-access-hjldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:48:42.844620 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.844564    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjldh\" (UniqueName: \"kubernetes.io/projected/c87d9701-09ae-4045-b67d-ffcea135e61d-kube-api-access-hjldh\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:48:42.844620 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.844586    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87d9701-09ae-4045-b67d-ffcea135e61d-tls-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:48:42.844620 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:42.844597    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87d9701-09ae-4045-b67d-ffcea135e61d-dshm\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:48:43.308357 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.308325    2577 generic.go:356] "Generic (PLEG): container finished" podID="c87d9701-09ae-4045-b67d-ffcea135e61d" containerID="6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13" exitCode=0
Jul 08 23:48:43.308802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.308366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"
Jul 08 23:48:43.308802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.308406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr" event={"ID":"c87d9701-09ae-4045-b67d-ffcea135e61d","Type":"ContainerDied","Data":"6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13"}
Jul 08 23:48:43.308802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.308449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr" event={"ID":"c87d9701-09ae-4045-b67d-ffcea135e61d","Type":"ContainerDied","Data":"5e1bef7d1766d64b56cb164b6babe3c96b5c270a336e474c2f3b0298ab7f8f59"}
Jul 08 23:48:43.308802 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.308467    2577 scope.go:117] "RemoveContainer" containerID="6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13"
Jul 08 23:48:43.317838 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.317823    2577 scope.go:117] "RemoveContainer" containerID="6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13"
Jul 08 23:48:43.318087 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:48:43.318064    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13\": container with ID starting with 6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13 not found: ID does not exist" containerID="6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13"
Jul 08 23:48:43.318176 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.318094    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13"} err="failed to get container status \"6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13\": rpc error: code = NotFound desc = could not find container \"6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13\": container with ID starting with 6b747d7aa9cf22b44254d396ad99d98d3d5be5a811eda2a979675ad98ddb0e13 not found: ID does not exist"
Jul 08 23:48:43.329389 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.329368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"]
Jul 08 23:48:43.333445 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.333427    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cd1865/sub-test-model-cd1865-kserve-676c8d6b8f-qlfjr"]
Jul 08 23:48:43.521352 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:48:43.521322    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c87d9701-09ae-4045-b67d-ffcea135e61d" path="/var/lib/kubelet/pods/c87d9701-09ae-4045-b67d-ffcea135e61d/volumes"
Jul 08 23:49:47.521215 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:47.521180    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"]
Jul 08 23:49:49.549960 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.549918    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762893232"]
Jul 08 23:49:49.550487 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.550466    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87d9701-09ae-4045-b67d-ffcea135e61d" containerName="main"
Jul 08 23:49:49.550575 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.550490    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87d9701-09ae-4045-b67d-ffcea135e61d" containerName="main"
Jul 08 23:49:49.550575 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.550570    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87d9701-09ae-4045-b67d-ffcea135e61d" containerName="main"
Jul 08 23:49:49.553574 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.553552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:49:49.556067 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.556045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w7p9k\""
Jul 08 23:49:49.558627 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.558598    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762893232"]
Jul 08 23:49:49.647091 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.647071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drp7\" (UniqueName: \"kubernetes.io/projected/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1-kube-api-access-4drp7\") pod \"test-curl-7245-139765762893232\" (UID: \"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:49:49.747430 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.747406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drp7\" (UniqueName: \"kubernetes.io/projected/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1-kube-api-access-4drp7\") pod \"test-curl-7245-139765762893232\" (UID: \"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:49:49.758352 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.758320    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drp7\" (UniqueName: \"kubernetes.io/projected/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1-kube-api-access-4drp7\") pod \"test-curl-7245-139765762893232\" (UID: \"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:49:49.863865 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.863843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:49:49.990578 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.990559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762893232"]
Jul 08 23:49:49.992689 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:49:49.992665    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2c2b9d9_1c20_4d6b_8801_c1a7767536b1.slice/crio-59df4687fba4b548ebc308d7d56fda7fcdd96c4e5172b1e72b03f267b5e928fb WatchSource:0}: Error finding container 59df4687fba4b548ebc308d7d56fda7fcdd96c4e5172b1e72b03f267b5e928fb: Status 404 returned error can't find the container with id 59df4687fba4b548ebc308d7d56fda7fcdd96c4e5172b1e72b03f267b5e928fb
Jul 08 23:49:49.993922 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:49.993905    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 23:49:50.627019 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:50.626986    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232" event={"ID":"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1","Type":"ContainerStarted","Data":"59df4687fba4b548ebc308d7d56fda7fcdd96c4e5172b1e72b03f267b5e928fb"}
Jul 08 23:49:51.632237 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:51.632206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232" event={"ID":"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1","Type":"ContainerStarted","Data":"b78d78b2f23228d777f07fda272dadfb4372031f87e528574cea278745300dc7"}
Jul 08 23:49:51.645950 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:49:51.645905    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232" podStartSLOduration=1.948881539 podStartE2EDuration="2.64588997s" totalImagesPullingTime="697.008431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:49:51.644889052 +0000 UTC m=+2540.636722361" watchObservedRunningTime="2026-07-08 23:49:51.64588997 +0000 UTC m=+2540.637723324"
Jul 08 23:50:01.678313 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:01.678279    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2c2b9d9-1c20-4d6b-8801-c1a7767536b1" containerID="b78d78b2f23228d777f07fda272dadfb4372031f87e528574cea278745300dc7" exitCode=28
Jul 08 23:50:01.678695 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:01.678337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232" event={"ID":"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1","Type":"ContainerDied","Data":"b78d78b2f23228d777f07fda272dadfb4372031f87e528574cea278745300dc7"}
Jul 08 23:50:02.540949 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.540913    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl" podUID="d9fbbbde-277f-4a0e-b61c-bc47bf351869" containerName="main" containerID="cri-o://6cf6d877689fd328054c6f8738f5dedf37a202a6c91108d99bcd122cdf14b073" gracePeriod=45
Jul 08 23:50:02.687654 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.687627    2577 generic.go:356] "Generic (PLEG): container finished" podID="d9fbbbde-277f-4a0e-b61c-bc47bf351869" containerID="6cf6d877689fd328054c6f8738f5dedf37a202a6c91108d99bcd122cdf14b073" exitCode=0
Jul 08 23:50:02.688065 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.687696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl" event={"ID":"d9fbbbde-277f-4a0e-b61c-bc47bf351869","Type":"ContainerDied","Data":"6cf6d877689fd328054c6f8738f5dedf37a202a6c91108d99bcd122cdf14b073"}
Jul 08 23:50:02.798484 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.798423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:50:02.835979 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.835961    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:50:02.869403 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.869377    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762893232"]
Jul 08 23:50:02.874512 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.874491    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762893232"]
Jul 08 23:50:02.944664 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-dshm\") pod \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") "
Jul 08 23:50:02.944774 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944671    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85dfh\" (UniqueName: \"kubernetes.io/projected/d9fbbbde-277f-4a0e-b61c-bc47bf351869-kube-api-access-85dfh\") pod \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") "
Jul 08 23:50:02.944774 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tmp-dir\") pod \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") "
Jul 08 23:50:02.944774 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-model-cache\") pod \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") "
Jul 08 23:50:02.944942 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-home\") pod \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") "
Jul 08 23:50:02.944942 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4drp7\" (UniqueName: \"kubernetes.io/projected/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1-kube-api-access-4drp7\") pod \"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1\" (UID: \"a2c2b9d9-1c20-4d6b-8801-c1a7767536b1\") "
Jul 08 23:50:02.944942 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.944935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tls-certs\") pod \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\" (UID: \"d9fbbbde-277f-4a0e-b61c-bc47bf351869\") "
Jul 08 23:50:02.945103 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.945002    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9fbbbde-277f-4a0e-b61c-bc47bf351869" (UID: "d9fbbbde-277f-4a0e-b61c-bc47bf351869"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:50:02.945103 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.945045    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9fbbbde-277f-4a0e-b61c-bc47bf351869" (UID: "d9fbbbde-277f-4a0e-b61c-bc47bf351869"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:50:02.945184 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.945105    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-home" (OuterVolumeSpecName: "home") pod "d9fbbbde-277f-4a0e-b61c-bc47bf351869" (UID: "d9fbbbde-277f-4a0e-b61c-bc47bf351869"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:50:02.945315 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.945290    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tmp-dir\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:02.945315 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.945316    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-model-cache\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:02.945486 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.945329    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-home\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:02.946686 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.946666    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-dshm" (OuterVolumeSpecName: "dshm") pod "d9fbbbde-277f-4a0e-b61c-bc47bf351869" (UID: "d9fbbbde-277f-4a0e-b61c-bc47bf351869"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 23:50:02.946892 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.946869    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1-kube-api-access-4drp7" (OuterVolumeSpecName: "kube-api-access-4drp7") pod "a2c2b9d9-1c20-4d6b-8801-c1a7767536b1" (UID: "a2c2b9d9-1c20-4d6b-8801-c1a7767536b1"). InnerVolumeSpecName "kube-api-access-4drp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:50:02.946892 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.946877    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fbbbde-277f-4a0e-b61c-bc47bf351869-kube-api-access-85dfh" (OuterVolumeSpecName: "kube-api-access-85dfh") pod "d9fbbbde-277f-4a0e-b61c-bc47bf351869" (UID: "d9fbbbde-277f-4a0e-b61c-bc47bf351869"). InnerVolumeSpecName "kube-api-access-85dfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:50:02.947011 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:02.946894    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9fbbbde-277f-4a0e-b61c-bc47bf351869" (UID: "d9fbbbde-277f-4a0e-b61c-bc47bf351869"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:50:03.012895 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.012869    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762892080"]
Jul 08 23:50:03.013205 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.013192    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9fbbbde-277f-4a0e-b61c-bc47bf351869" containerName="main"
Jul 08 23:50:03.013247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.013207    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9fbbbde-277f-4a0e-b61c-bc47bf351869" containerName="main"
Jul 08 23:50:03.013247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.013230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c2b9d9-1c20-4d6b-8801-c1a7767536b1" containerName="test-curl-7245-139765762893232"
Jul 08 23:50:03.013247 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.013237    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c2b9d9-1c20-4d6b-8801-c1a7767536b1" containerName="test-curl-7245-139765762893232"
Jul 08 23:50:03.013345 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.013295    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9fbbbde-277f-4a0e-b61c-bc47bf351869" containerName="main"
Jul 08 23:50:03.013345 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.013304    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c2b9d9-1c20-4d6b-8801-c1a7767536b1" containerName="test-curl-7245-139765762893232"
Jul 08 23:50:03.016400 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.016384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:03.021973 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.021946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762892080"]
Jul 08 23:50:03.046006 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.045988    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9fbbbde-277f-4a0e-b61c-bc47bf351869-dshm\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:03.046006 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.046008    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85dfh\" (UniqueName: \"kubernetes.io/projected/d9fbbbde-277f-4a0e-b61c-bc47bf351869-kube-api-access-85dfh\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:03.046137 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.046020    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drp7\" (UniqueName: \"kubernetes.io/projected/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1-kube-api-access-4drp7\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:03.046137 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.046029    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9fbbbde-277f-4a0e-b61c-bc47bf351869-tls-certs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:03.147041 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.147023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktdpx\" (UniqueName: \"kubernetes.io/projected/d997798e-9262-4fd7-81a1-0c664ca3df99-kube-api-access-ktdpx\") pod \"test-curl-7245-139765762892080\" (UID: \"d997798e-9262-4fd7-81a1-0c664ca3df99\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:03.247600 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.247571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktdpx\" (UniqueName: \"kubernetes.io/projected/d997798e-9262-4fd7-81a1-0c664ca3df99-kube-api-access-ktdpx\") pod \"test-curl-7245-139765762892080\" (UID: \"d997798e-9262-4fd7-81a1-0c664ca3df99\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:03.255233 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.255216    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktdpx\" (UniqueName: \"kubernetes.io/projected/d997798e-9262-4fd7-81a1-0c664ca3df99-kube-api-access-ktdpx\") pod \"test-curl-7245-139765762892080\" (UID: \"d997798e-9262-4fd7-81a1-0c664ca3df99\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:03.325532 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.325503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:03.451869 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.451842    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762892080"]
Jul 08 23:50:03.453820 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:50:03.453796    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd997798e_9262_4fd7_81a1_0c664ca3df99.slice/crio-9fd485eb760b8f7d4b1bf99dc4ebdc98cea849fc93eb5cefde1d1317fd3725f5 WatchSource:0}: Error finding container 9fd485eb760b8f7d4b1bf99dc4ebdc98cea849fc93eb5cefde1d1317fd3725f5: Status 404 returned error can't find the container with id 9fd485eb760b8f7d4b1bf99dc4ebdc98cea849fc93eb5cefde1d1317fd3725f5
Jul 08 23:50:03.519512 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.519489    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c2b9d9-1c20-4d6b-8801-c1a7767536b1" path="/var/lib/kubelet/pods/a2c2b9d9-1c20-4d6b-8801-c1a7767536b1/volumes"
Jul 08 23:50:03.694198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.694141    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"
Jul 08 23:50:03.694198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.694149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl" event={"ID":"d9fbbbde-277f-4a0e-b61c-bc47bf351869","Type":"ContainerDied","Data":"a372fc55839aa18e83a935a0eec244a59d20b46bc2fa8055975a1de712b4e28f"}
Jul 08 23:50:03.694198 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.694191    2577 scope.go:117] "RemoveContainer" containerID="6cf6d877689fd328054c6f8738f5dedf37a202a6c91108d99bcd122cdf14b073"
Jul 08 23:50:03.695795 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.695752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762893232"
Jul 08 23:50:03.697344 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.697321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080" event={"ID":"d997798e-9262-4fd7-81a1-0c664ca3df99","Type":"ContainerStarted","Data":"9fd485eb760b8f7d4b1bf99dc4ebdc98cea849fc93eb5cefde1d1317fd3725f5"}
Jul 08 23:50:03.703093 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.703072    2577 scope.go:117] "RemoveContainer" containerID="b78d78b2f23228d777f07fda272dadfb4372031f87e528574cea278745300dc7"
Jul 08 23:50:03.710602 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.710584    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"]
Jul 08 23:50:03.714404 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:03.714385    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-310c89/rate-test-model-310c89-kserve-84d6588c5f-h46gl"]
Jul 08 23:50:04.706644 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:04.706602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080" event={"ID":"d997798e-9262-4fd7-81a1-0c664ca3df99","Type":"ContainerStarted","Data":"14491c64438b9145214fca33ddbd1c606e4717a757b303593a6bd59d8afbc32c"}
Jul 08 23:50:04.721142 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:04.721085    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080" podStartSLOduration=1.42698976 podStartE2EDuration="1.72106945s" totalImagesPullingTime="294.07969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:50:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:50:04.720794683 +0000 UTC m=+2553.712627992" watchObservedRunningTime="2026-07-08 23:50:04.72106945 +0000 UTC m=+2553.712902755"
Jul 08 23:50:05.520218 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:05.520180    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9fbbbde-277f-4a0e-b61c-bc47bf351869" path="/var/lib/kubelet/pods/d9fbbbde-277f-4a0e-b61c-bc47bf351869/volumes"
Jul 08 23:50:14.755569 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:14.755536    2577 generic.go:356] "Generic (PLEG): container finished" podID="d997798e-9262-4fd7-81a1-0c664ca3df99" containerID="14491c64438b9145214fca33ddbd1c606e4717a757b303593a6bd59d8afbc32c" exitCode=28
Jul 08 23:50:14.755951 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:14.755610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080" event={"ID":"d997798e-9262-4fd7-81a1-0c664ca3df99","Type":"ContainerDied","Data":"14491c64438b9145214fca33ddbd1c606e4717a757b303593a6bd59d8afbc32c"}
Jul 08 23:50:15.893014 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:15.892986    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:15.925579 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:15.925545    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762892080"]
Jul 08 23:50:15.930268 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:15.930242    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762892080"]
Jul 08 23:50:16.047871 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.047813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktdpx\" (UniqueName: \"kubernetes.io/projected/d997798e-9262-4fd7-81a1-0c664ca3df99-kube-api-access-ktdpx\") pod \"d997798e-9262-4fd7-81a1-0c664ca3df99\" (UID: \"d997798e-9262-4fd7-81a1-0c664ca3df99\") "
Jul 08 23:50:16.049745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.049720    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d997798e-9262-4fd7-81a1-0c664ca3df99-kube-api-access-ktdpx" (OuterVolumeSpecName: "kube-api-access-ktdpx") pod "d997798e-9262-4fd7-81a1-0c664ca3df99" (UID: "d997798e-9262-4fd7-81a1-0c664ca3df99"). InnerVolumeSpecName "kube-api-access-ktdpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:50:16.100910 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.100881    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762983952"]
Jul 08 23:50:16.101258 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.101241    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d997798e-9262-4fd7-81a1-0c664ca3df99" containerName="test-curl-7245-139765762892080"
Jul 08 23:50:16.101339 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.101261    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d997798e-9262-4fd7-81a1-0c664ca3df99" containerName="test-curl-7245-139765762892080"
Jul 08 23:50:16.101392 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.101378    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d997798e-9262-4fd7-81a1-0c664ca3df99" containerName="test-curl-7245-139765762892080"
Jul 08 23:50:16.104266 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.104249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:16.108322 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.108303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762983952"]
Jul 08 23:50:16.149001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.148979    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktdpx\" (UniqueName: \"kubernetes.io/projected/d997798e-9262-4fd7-81a1-0c664ca3df99-kube-api-access-ktdpx\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:16.250310 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.250288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9474d\" (UniqueName: \"kubernetes.io/projected/e826f5a9-0c20-4ee9-89f2-2f414c5083bf-kube-api-access-9474d\") pod \"test-curl-7245-139765762983952\" (UID: \"e826f5a9-0c20-4ee9-89f2-2f414c5083bf\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:16.350915 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.350857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9474d\" (UniqueName: \"kubernetes.io/projected/e826f5a9-0c20-4ee9-89f2-2f414c5083bf-kube-api-access-9474d\") pod \"test-curl-7245-139765762983952\" (UID: \"e826f5a9-0c20-4ee9-89f2-2f414c5083bf\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:16.359999 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.359968    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9474d\" (UniqueName: \"kubernetes.io/projected/e826f5a9-0c20-4ee9-89f2-2f414c5083bf-kube-api-access-9474d\") pod \"test-curl-7245-139765762983952\" (UID: \"e826f5a9-0c20-4ee9-89f2-2f414c5083bf\") " pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:16.413281 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.413260    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:16.538310 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.538284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762983952"]
Jul 08 23:50:16.540461 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:50:16.540437    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode826f5a9_0c20_4ee9_89f2_2f414c5083bf.slice/crio-d7a7902b60d52a8c06853952826ca428365a739090a0eb3b237db337821e9da2 WatchSource:0}: Error finding container d7a7902b60d52a8c06853952826ca428365a739090a0eb3b237db337821e9da2: Status 404 returned error can't find the container with id d7a7902b60d52a8c06853952826ca428365a739090a0eb3b237db337821e9da2
Jul 08 23:50:16.764899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.764873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:16.764899 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.764881    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd485eb760b8f7d4b1bf99dc4ebdc98cea849fc93eb5cefde1d1317fd3725f5"
Jul 08 23:50:16.766055 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:16.766031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952" event={"ID":"e826f5a9-0c20-4ee9-89f2-2f414c5083bf","Type":"ContainerStarted","Data":"d7a7902b60d52a8c06853952826ca428365a739090a0eb3b237db337821e9da2"}
Jul 08 23:50:16.775219 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:50:16.775195    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7245-139765762892080\" is forbidden: User \"system:node:ip-10-0-130-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-115.ec2.internal' and this object" podUID="d997798e-9262-4fd7-81a1-0c664ca3df99" pod="odh-ai-gateway-infra/test-curl-7245-139765762892080"
Jul 08 23:50:17.520399 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:17.520368    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d997798e-9262-4fd7-81a1-0c664ca3df99" path="/var/lib/kubelet/pods/d997798e-9262-4fd7-81a1-0c664ca3df99/volumes"
Jul 08 23:50:17.771664 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:17.771588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952" event={"ID":"e826f5a9-0c20-4ee9-89f2-2f414c5083bf","Type":"ContainerStarted","Data":"f4af9d035ad46265b664ed1a72a5c089906730519cf235cc2f3f3c9ad66fd616"}
Jul 08 23:50:17.786710 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:17.786657    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952" podStartSLOduration=1.4593056770000001 podStartE2EDuration="1.786641721s" totalImagesPullingTime="327.336044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 23:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:50:17.785843016 +0000 UTC m=+2566.777676323" watchObservedRunningTime="2026-07-08 23:50:17.786641721 +0000 UTC m=+2566.778475028"
Jul 08 23:50:27.822374 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:27.822334    2577 generic.go:356] "Generic (PLEG): container finished" podID="e826f5a9-0c20-4ee9-89f2-2f414c5083bf" containerID="f4af9d035ad46265b664ed1a72a5c089906730519cf235cc2f3f3c9ad66fd616" exitCode=28
Jul 08 23:50:27.822816 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:27.822408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952" event={"ID":"e826f5a9-0c20-4ee9-89f2-2f414c5083bf","Type":"ContainerDied","Data":"f4af9d035ad46265b664ed1a72a5c089906730519cf235cc2f3f3c9ad66fd616"}
Jul 08 23:50:28.962740 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:28.962715    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:28.993000 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:28.992973    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762983952"]
Jul 08 23:50:28.995472 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:28.995449    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/test-curl-7245-139765762983952"]
Jul 08 23:50:29.043863 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.043840    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9474d\" (UniqueName: \"kubernetes.io/projected/e826f5a9-0c20-4ee9-89f2-2f414c5083bf-kube-api-access-9474d\") pod \"e826f5a9-0c20-4ee9-89f2-2f414c5083bf\" (UID: \"e826f5a9-0c20-4ee9-89f2-2f414c5083bf\") "
Jul 08 23:50:29.045841 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.045816    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e826f5a9-0c20-4ee9-89f2-2f414c5083bf-kube-api-access-9474d" (OuterVolumeSpecName: "kube-api-access-9474d") pod "e826f5a9-0c20-4ee9-89f2-2f414c5083bf" (UID: "e826f5a9-0c20-4ee9-89f2-2f414c5083bf"). InnerVolumeSpecName "kube-api-access-9474d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:50:29.145176 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.145157    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9474d\" (UniqueName: \"kubernetes.io/projected/e826f5a9-0c20-4ee9-89f2-2f414c5083bf-kube-api-access-9474d\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:29.520105 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.520028    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e826f5a9-0c20-4ee9-89f2-2f414c5083bf" path="/var/lib/kubelet/pods/e826f5a9-0c20-4ee9-89f2-2f414c5083bf/volumes"
Jul 08 23:50:29.834294 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.834226    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/test-curl-7245-139765762983952"
Jul 08 23:50:29.834424 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.834225    2577 scope.go:117] "RemoveContainer" containerID="f4af9d035ad46265b664ed1a72a5c089906730519cf235cc2f3f3c9ad66fd616"
Jul 08 23:50:29.993604 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.993572    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"]
Jul 08 23:50:29.994053 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:29.993924    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" podUID="02471e12-5645-4d14-9e7e-56978b2a8f66" containerName="maas-api" containerID="cri-o://e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab" gracePeriod=30
Jul 08 23:50:30.243784 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.243759    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:50:30.355743 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.355698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb6cs\" (UniqueName: \"kubernetes.io/projected/02471e12-5645-4d14-9e7e-56978b2a8f66-kube-api-access-zb6cs\") pod \"02471e12-5645-4d14-9e7e-56978b2a8f66\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") "
Jul 08 23:50:30.355870 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.355833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls\") pod \"02471e12-5645-4d14-9e7e-56978b2a8f66\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") "
Jul 08 23:50:30.355870 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.355861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02471e12-5645-4d14-9e7e-56978b2a8f66-openshift-service-ca\") pod \"02471e12-5645-4d14-9e7e-56978b2a8f66\" (UID: \"02471e12-5645-4d14-9e7e-56978b2a8f66\") "
Jul 08 23:50:30.356231 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.356208    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02471e12-5645-4d14-9e7e-56978b2a8f66-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02471e12-5645-4d14-9e7e-56978b2a8f66" (UID: "02471e12-5645-4d14-9e7e-56978b2a8f66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:50:30.357626 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.357603    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02471e12-5645-4d14-9e7e-56978b2a8f66-kube-api-access-zb6cs" (OuterVolumeSpecName: "kube-api-access-zb6cs") pod "02471e12-5645-4d14-9e7e-56978b2a8f66" (UID: "02471e12-5645-4d14-9e7e-56978b2a8f66"). InnerVolumeSpecName "kube-api-access-zb6cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:50:30.357745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.357726    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02471e12-5645-4d14-9e7e-56978b2a8f66" (UID: "02471e12-5645-4d14-9e7e-56978b2a8f66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:50:30.456771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.456740    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6cs\" (UniqueName: \"kubernetes.io/projected/02471e12-5645-4d14-9e7e-56978b2a8f66-kube-api-access-zb6cs\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:30.456771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.456771    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02471e12-5645-4d14-9e7e-56978b2a8f66-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:30.456771 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.456780    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02471e12-5645-4d14-9e7e-56978b2a8f66-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:30.842214 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.842183    2577 generic.go:356] "Generic (PLEG): container finished" podID="02471e12-5645-4d14-9e7e-56978b2a8f66" containerID="e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab" exitCode=0
Jul 08 23:50:30.842335 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.842217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" event={"ID":"02471e12-5645-4d14-9e7e-56978b2a8f66","Type":"ContainerDied","Data":"e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab"}
Jul 08 23:50:30.842335 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.842235    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"
Jul 08 23:50:30.842335 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.842251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk" event={"ID":"02471e12-5645-4d14-9e7e-56978b2a8f66","Type":"ContainerDied","Data":"a6a426bf800ded70be75674e2f939b783113a34168878d6fa0ded1721728acf4"}
Jul 08 23:50:30.842335 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.842273    2577 scope.go:117] "RemoveContainer" containerID="e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab"
Jul 08 23:50:30.850374 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.850340    2577 scope.go:117] "RemoveContainer" containerID="e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab"
Jul 08 23:50:30.850658 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:50:30.850632    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab\": container with ID starting with e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab not found: ID does not exist" containerID="e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab"
Jul 08 23:50:30.850738 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.850668    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab"} err="failed to get container status \"e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab\": rpc error: code = NotFound desc = could not find container \"e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab\": container with ID starting with e0587a0818daf741d9a28ad49a5edf7dc8f64cda58c5b14f21be13f4e06ffbab not found: ID does not exist"
Jul 08 23:50:30.863038 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.863011    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"]
Jul 08 23:50:30.867695 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:30.867676    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-310c89-867bcfd85d-wd5bk"]
Jul 08 23:50:31.522111 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:31.522081    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02471e12-5645-4d14-9e7e-56978b2a8f66" path="/var/lib/kubelet/pods/02471e12-5645-4d14-9e7e-56978b2a8f66/volumes"
Jul 08 23:50:36.959667 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:36.959636    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"]
Jul 08 23:50:36.960244 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:36.959855    2577 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw" podUID="97ac65b5-c4b9-4916-9a7f-64c692bf249a" containerName="maas-api" containerID="cri-o://4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b" gracePeriod=30
Jul 08 23:50:37.406409 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.406384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:50:37.508520 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.508433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6kkq\" (UniqueName: \"kubernetes.io/projected/97ac65b5-c4b9-4916-9a7f-64c692bf249a-kube-api-access-b6kkq\") pod \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") "
Jul 08 23:50:37.508520 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.508489    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac65b5-c4b9-4916-9a7f-64c692bf249a-openshift-service-ca\") pod \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") "
Jul 08 23:50:37.508520 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.508530    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97ac65b5-c4b9-4916-9a7f-64c692bf249a-maas-api-tls\") pod \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\" (UID: \"97ac65b5-c4b9-4916-9a7f-64c692bf249a\") "
Jul 08 23:50:37.508876 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.508852    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ac65b5-c4b9-4916-9a7f-64c692bf249a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97ac65b5-c4b9-4916-9a7f-64c692bf249a" (UID: "97ac65b5-c4b9-4916-9a7f-64c692bf249a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 23:50:37.510592 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.510569    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ac65b5-c4b9-4916-9a7f-64c692bf249a-kube-api-access-b6kkq" (OuterVolumeSpecName: "kube-api-access-b6kkq") pod "97ac65b5-c4b9-4916-9a7f-64c692bf249a" (UID: "97ac65b5-c4b9-4916-9a7f-64c692bf249a"). InnerVolumeSpecName "kube-api-access-b6kkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 23:50:37.510682 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.510614    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ac65b5-c4b9-4916-9a7f-64c692bf249a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97ac65b5-c4b9-4916-9a7f-64c692bf249a" (UID: "97ac65b5-c4b9-4916-9a7f-64c692bf249a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 23:50:37.609526 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.609505    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97ac65b5-c4b9-4916-9a7f-64c692bf249a-maas-api-tls\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:37.609526 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.609525    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kkq\" (UniqueName: \"kubernetes.io/projected/97ac65b5-c4b9-4916-9a7f-64c692bf249a-kube-api-access-b6kkq\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:37.609656 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.609535    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97ac65b5-c4b9-4916-9a7f-64c692bf249a-openshift-service-ca\") on node \"ip-10-0-130-115.ec2.internal\" DevicePath \"\""
Jul 08 23:50:37.878621 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.878596    2577 generic.go:356] "Generic (PLEG): container finished" podID="97ac65b5-c4b9-4916-9a7f-64c692bf249a" containerID="4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b" exitCode=0
Jul 08 23:50:37.878754 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.878623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw" event={"ID":"97ac65b5-c4b9-4916-9a7f-64c692bf249a","Type":"ContainerDied","Data":"4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b"}
Jul 08 23:50:37.878754 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.878643    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"
Jul 08 23:50:37.878754 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.878680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw" event={"ID":"97ac65b5-c4b9-4916-9a7f-64c692bf249a","Type":"ContainerDied","Data":"bd0de1b65046749111f8f68f2e70fe6e7930e79edc0921a5e4677ce4467ef6f1"}
Jul 08 23:50:37.878754 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.878724    2577 scope.go:117] "RemoveContainer" containerID="4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b"
Jul 08 23:50:37.887078 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.887062    2577 scope.go:117] "RemoveContainer" containerID="4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b"
Jul 08 23:50:37.887357 ip-10-0-130-115 kubenswrapper[2577]: E0708 23:50:37.887334    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b\": container with ID starting with 4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b not found: ID does not exist" containerID="4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b"
Jul 08 23:50:37.887411 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.887371    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b"} err="failed to get container status \"4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b\": rpc error: code = NotFound desc = could not find container \"4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b\": container with ID starting with 4d17996405a0fef68fcb74eb5f82ba272ba2b454c29debb611ebdbcb6dc9b27b not found: ID does not exist"
Jul 08 23:50:37.894199 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.894181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"]
Jul 08 23:50:37.900178 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:37.900157    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-cd1865-f68cb554d-4g6mw"]
Jul 08 23:50:39.520609 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:39.520581    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ac65b5-c4b9-4916-9a7f-64c692bf249a" path="/var/lib/kubelet/pods/97ac65b5-c4b9-4916-9a7f-64c692bf249a/volumes"
Jul 08 23:50:43.529212 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:43.529174    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-786479bc9f-rn482_bbcc4057-726f-4b4e-be16-d95d620d748a/authorino/0.log"
Jul 08 23:50:48.584250 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:48.584211    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7464f4d4d-zzkmx_9b95ca82-e911-482a-ae68-d49d2c641dc1/manager/0.log"
Jul 08 23:50:48.815064 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:48.815034    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-btndx_20bfb057-f2e5-4ee8-a0eb-ec32f856c558/manager/0.log"
Jul 08 23:50:48.926629 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:48.926610    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b86895794-8hwqb_c35da1e3-ce78-42c2-b79f-aa7769c9aa30/manager/0.log"
Jul 08 23:50:48.932803 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:48.932782    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b86895794-8hwqb_c35da1e3-ce78-42c2-b79f-aa7769c9aa30/copy-manifests/0.log"
Jul 08 23:50:49.541957 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:49.541931    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5f6c8885dd-pjlcp_3fabbb9e-69a6-4168-a2e9-2a467806e501/maas-api/0.log"
Jul 08 23:50:50.999165 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:50.999135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-786479bc9f-rn482_bbcc4057-726f-4b4e-be16-d95d620d748a/authorino/0.log"
Jul 08 23:50:52.556805 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:52.556771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b8998bd98-k9259_1ca08daf-44f5-406d-aa54-dc36507f0b47/kube-auth-proxy/0.log"
Jul 08 23:50:52.791261 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:52.791232    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8t7tb_adef2d3c-bde5-4788-8fa8-dc967afb75ea/payload-pre-processing/0.log"
Jul 08 23:50:53.587125 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:53.587042    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tgfd4_be3c2277-789b-45b6-bdbd-0a2f28f2170e/main/0.log"
Jul 08 23:50:53.700427 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:50:53.700406    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbw4cgx_d5e4b150-a17f-41de-b111-adf33fa6ff15/main/0.log"
Jul 08 23:51:01.098315 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:01.098285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x4psn_dbf63736-facd-4dd4-ac3e-418bc5356176/global-pull-secret-syncer/0.log"
Jul 08 23:51:01.211115 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:01.211083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zwhlx_eb5b38f8-7c7a-4f3e-bbe7-78a70800814a/konnectivity-agent/0.log"
Jul 08 23:51:01.231454 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:01.231433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-115.ec2.internal_4b46bf19a4c8c74cc56a4f755de60986/haproxy/0.log"
Jul 08 23:51:06.708122 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:06.708089    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-786479bc9f-rn482_bbcc4057-726f-4b4e-be16-d95d620d748a/authorino/0.log"
Jul 08 23:51:08.704850 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.704771    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/alertmanager/0.log"
Jul 08 23:51:08.728183 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.728158    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/config-reloader/0.log"
Jul 08 23:51:08.749685 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.749660    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/kube-rbac-proxy-web/0.log"
Jul 08 23:51:08.771641 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.771621    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/kube-rbac-proxy/0.log"
Jul 08 23:51:08.792217 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.792188    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/kube-rbac-proxy-metric/0.log"
Jul 08 23:51:08.814278 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.814260    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/prom-label-proxy/0.log"
Jul 08 23:51:08.835143 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:08.835122    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_989e9857-e6b5-4ca9-a2af-721b899994ae/init-config-reloader/0.log"
Jul 08 23:51:09.031225 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.031168    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b5zbg_f45751c2-6054-46bc-8966-ddf9839896f7/node-exporter/0.log"
Jul 08 23:51:09.053479 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.053464    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b5zbg_f45751c2-6054-46bc-8966-ddf9839896f7/kube-rbac-proxy/0.log"
Jul 08 23:51:09.088496 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.088478    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b5zbg_f45751c2-6054-46bc-8966-ddf9839896f7/init-textfile/0.log"
Jul 08 23:51:09.268430 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.268412    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-6rvtp_c0607942-de2c-40d5-ac35-65d7e30a200e/kube-rbac-proxy-main/0.log"
Jul 08 23:51:09.288636 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.288569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-6rvtp_c0607942-de2c-40d5-ac35-65d7e30a200e/kube-rbac-proxy-self/0.log"
Jul 08 23:51:09.311959 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.311936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-6rvtp_c0607942-de2c-40d5-ac35-65d7e30a200e/openshift-state-metrics/0.log"
Jul 08 23:51:09.503612 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.503585    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qrhhp_b65036aa-8028-4d0a-a38d-1bff65c5e464/prometheus-operator/0.log"
Jul 08 23:51:09.520612 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:09.520589    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-qrhhp_b65036aa-8028-4d0a-a38d-1bff65c5e464/kube-rbac-proxy/0.log"
Jul 08 23:51:11.038055 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038016    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"]
Jul 08 23:51:11.038576 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038552    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ac65b5-c4b9-4916-9a7f-64c692bf249a" containerName="maas-api"
Jul 08 23:51:11.038657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038579    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ac65b5-c4b9-4916-9a7f-64c692bf249a" containerName="maas-api"
Jul 08 23:51:11.038657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038594    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e826f5a9-0c20-4ee9-89f2-2f414c5083bf" containerName="test-curl-7245-139765762983952"
Jul 08 23:51:11.038657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038604    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e826f5a9-0c20-4ee9-89f2-2f414c5083bf" containerName="test-curl-7245-139765762983952"
Jul 08 23:51:11.038657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038615    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02471e12-5645-4d14-9e7e-56978b2a8f66" containerName="maas-api"
Jul 08 23:51:11.038657 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038623    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02471e12-5645-4d14-9e7e-56978b2a8f66" containerName="maas-api"
Jul 08 23:51:11.039002 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038767    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e826f5a9-0c20-4ee9-89f2-2f414c5083bf" containerName="test-curl-7245-139765762983952"
Jul 08 23:51:11.039002 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038782    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02471e12-5645-4d14-9e7e-56978b2a8f66" containerName="maas-api"
Jul 08 23:51:11.039002 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.038791    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ac65b5-c4b9-4916-9a7f-64c692bf249a" containerName="maas-api"
Jul 08 23:51:11.041815 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.041792    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.044525 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.044502    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bcsht\"/\"kube-root-ca.crt\""
Jul 08 23:51:11.044525 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.044506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bcsht\"/\"default-dockercfg-twmx9\""
Jul 08 23:51:11.045382 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.045364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bcsht\"/\"openshift-service-ca.crt\""
Jul 08 23:51:11.050272 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.050253    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"]
Jul 08 23:51:11.163116 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.163093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-proc\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.163223 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.163137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-podres\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.163223 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.163155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-lib-modules\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.163223 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.163172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82cv\" (UniqueName: \"kubernetes.io/projected/23c53859-8794-49c4-bd65-1d20cd82ec73-kube-api-access-h82cv\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.163333 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.163238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-sys\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264520 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-proc\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264596 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-podres\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264596 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-lib-modules\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264596 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82cv\" (UniqueName: \"kubernetes.io/projected/23c53859-8794-49c4-bd65-1d20cd82ec73-kube-api-access-h82cv\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264617    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-proc\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-sys\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-podres\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264667    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-sys\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.264745 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.264690    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23c53859-8794-49c4-bd65-1d20cd82ec73-lib-modules\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.272386 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.272364    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82cv\" (UniqueName: \"kubernetes.io/projected/23c53859-8794-49c4-bd65-1d20cd82ec73-kube-api-access-h82cv\") pod \"perf-node-gather-daemonset-prfvv\" (UID: \"23c53859-8794-49c4-bd65-1d20cd82ec73\") " pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.353819 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.353770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:11.482001 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:11.481976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"]
Jul 08 23:51:11.484200 ip-10-0-130-115 kubenswrapper[2577]: W0708 23:51:11.484174    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23c53859_8794_49c4_bd65_1d20cd82ec73.slice/crio-8cfa0b298f74fe90c83ecc3b6d0adbf1b53480d6ddd5a0dfd538ea4da3ac3744 WatchSource:0}: Error finding container 8cfa0b298f74fe90c83ecc3b6d0adbf1b53480d6ddd5a0dfd538ea4da3ac3744: Status 404 returned error can't find the container with id 8cfa0b298f74fe90c83ecc3b6d0adbf1b53480d6ddd5a0dfd538ea4da3ac3744
Jul 08 23:51:12.017053 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:12.017022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-lsd2g_f3ad60a8-7f4d-46d0-ad44-49fd54bec888/download-server/0.log"
Jul 08 23:51:12.050098 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:12.050069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv" event={"ID":"23c53859-8794-49c4-bd65-1d20cd82ec73","Type":"ContainerStarted","Data":"9b31ce08eb351ecac2610b8726ae677f280dadc009124398072c07142c930689"}
Jul 08 23:51:12.050447 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:12.050116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv" event={"ID":"23c53859-8794-49c4-bd65-1d20cd82ec73","Type":"ContainerStarted","Data":"8cfa0b298f74fe90c83ecc3b6d0adbf1b53480d6ddd5a0dfd538ea4da3ac3744"}
Jul 08 23:51:12.050447 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:12.050146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:12.067736 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:12.067664    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv" podStartSLOduration=1.067647733 podStartE2EDuration="1.067647733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 23:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 23:51:12.065427877 +0000 UTC m=+2621.057261184" watchObservedRunningTime="2026-07-08 23:51:12.067647733 +0000 UTC m=+2621.059481040"
Jul 08 23:51:13.597238 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:13.597205    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7r4qc_32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0/dns/0.log"
Jul 08 23:51:13.617354 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:13.617334    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7r4qc_32ac82dc-4d55-486f-a9f9-6ae2a2a62bf0/kube-rbac-proxy/0.log"
Jul 08 23:51:13.687289 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:13.687263    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hj86f_2c5b15c7-69bb-496a-9080-bb4ac43ae4fb/dns-node-resolver/0.log"
Jul 08 23:51:14.305196 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:14.305167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74f86749cc-7qlw4_b8c8c860-b7bd-448e-9828-2f4c877269f1/registry/0.log"
Jul 08 23:51:14.390416 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:14.390394    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-45jkz_88b0968b-f35e-48d1-9e4a-62d157eda27b/node-ca/0.log"
Jul 08 23:51:15.518304 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:15.518268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b8998bd98-k9259_1ca08daf-44f5-406d-aa54-dc36507f0b47/kube-auth-proxy/0.log"
Jul 08 23:51:15.650250 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:15.650216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8t7tb_adef2d3c-bde5-4788-8fa8-dc967afb75ea/payload-pre-processing/0.log"
Jul 08 23:51:16.412427 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:16.412395    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xzsqd_01ba5fc2-4c03-47c3-9bb6-54682cd7d25f/serve-healthcheck-canary/0.log"
Jul 08 23:51:16.952507 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:16.952474    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9wx5_75a531ef-9012-4bb3-ba32-089218afc067/kube-rbac-proxy/0.log"
Jul 08 23:51:16.972687 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:16.972667    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9wx5_75a531ef-9012-4bb3-ba32-089218afc067/exporter/0.log"
Jul 08 23:51:16.993977 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:16.993951    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9wx5_75a531ef-9012-4bb3-ba32-089218afc067/extractor/0.log"
Jul 08 23:51:18.065472 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:18.065444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bcsht/perf-node-gather-daemonset-prfvv"
Jul 08 23:51:19.442159 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:19.442131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7464f4d4d-zzkmx_9b95ca82-e911-482a-ae68-d49d2c641dc1/manager/0.log"
Jul 08 23:51:19.571242 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:19.571211    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-btndx_20bfb057-f2e5-4ee8-a0eb-ec32f856c558/manager/0.log"
Jul 08 23:51:19.593924 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:19.593900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b86895794-8hwqb_c35da1e3-ce78-42c2-b79f-aa7769c9aa30/manager/0.log"
Jul 08 23:51:19.615327 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:19.615309    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b86895794-8hwqb_c35da1e3-ce78-42c2-b79f-aa7769c9aa30/copy-manifests/0.log"
Jul 08 23:51:21.189239 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:21.189144    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56bdb7f565-9c2bw_b3964231-a862-4bdf-af43-029ad615875d/manager/0.log"
Jul 08 23:51:21.889290 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:21.889261    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b8998bd98-k9259_1ca08daf-44f5-406d-aa54-dc36507f0b47/kube-auth-proxy/0.log"
Jul 08 23:51:21.996511 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:21.996469    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8t7tb_adef2d3c-bde5-4788-8fa8-dc967afb75ea/payload-pre-processing/0.log"
Jul 08 23:51:23.321791 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:23.321757    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b8998bd98-k9259_1ca08daf-44f5-406d-aa54-dc36507f0b47/kube-auth-proxy/0.log"
Jul 08 23:51:23.446943 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:23.446916    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-8t7tb_adef2d3c-bde5-4788-8fa8-dc967afb75ea/payload-pre-processing/0.log"
Jul 08 23:51:27.575387 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.575360    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/kube-multus-additional-cni-plugins/0.log"
Jul 08 23:51:27.606973 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.606947    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/egress-router-binary-copy/0.log"
Jul 08 23:51:27.628794 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.628774    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/cni-plugins/0.log"
Jul 08 23:51:27.648320 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.648304    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/bond-cni-plugin/0.log"
Jul 08 23:51:27.667647 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.667623    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/routeoverride-cni/0.log"
Jul 08 23:51:27.687040 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.687022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/whereabouts-cni-bincopy/0.log"
Jul 08 23:51:27.705986 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:27.705968    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vd9d9_753217f0-e74b-40cf-a22f-6ddb1e4af404/whereabouts-cni/0.log"
Jul 08 23:51:28.093212 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:28.093183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gzvzj_476b5c69-63b7-4a24-8843-a0a97efe69c8/kube-multus/0.log"
Jul 08 23:51:28.137267 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:28.137244    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ftnp8_e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6/network-metrics-daemon/0.log"
Jul 08 23:51:28.155635 ip-10-0-130-115 kubenswrapper[2577]: I0708 23:51:28.155612    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ftnp8_e90fa5b1-0bc9-4715-8f33-2f9eb8c421c6/kube-rbac-proxy/0.log"