Jul 13 20:12:14.257107 ip-10-0-128-62 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 13 20:12:14.257119 ip-10-0-128-62 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 13 20:12:14.257126 ip-10-0-128-62 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 13 20:12:14.257367 ip-10-0-128-62 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 13 20:12:25.813222 ip-10-0-128-62 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 13 20:12:25.813240 ip-10-0-128-62 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ee5c74aedf574e6c8429ed36539ed888 --
Jul 13 20:14:25.977685 ip-10-0-128-62 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 20:14:26.395326 ip-10-0-128-62 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:14:26.395326 ip-10-0-128-62 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 20:14:26.395326 ip-10-0-128-62 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:14:26.395326 ip-10-0-128-62 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 20:14:26.395326 ip-10-0-128-62 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:14:26.397499 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.397380    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 20:14:26.401765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401743    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:14:26.401765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401765    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:14:26.401765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401770    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:14:26.401765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401774    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401778    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401783    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401786    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401790    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401795    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401798    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401801    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401805    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401809    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401813    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401816    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401821    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401824    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401828    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401833    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401837    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401841    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401845    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:14:26.402021 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401849    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401852    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401856    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401860    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401864    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401868    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401872    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401876    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401880    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401885    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401888    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401892    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401896    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401902    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401906    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401910    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401916    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401920    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401925    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401929    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:14:26.402820 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401933    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401937    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401941    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401945    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401949    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401953    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401959    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401963    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401967    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401971    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401975    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401979    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401983    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401987    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401992    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.401999    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402005    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402010    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402014    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402018    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:14:26.403675 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402022    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402026    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402030    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402035    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402039    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402044    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402048    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402053    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402058    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402063    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402067    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402071    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402075    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402080    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402084    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402088    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402092    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402097    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402101    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402105    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:14:26.404525 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402109    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402116    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402121    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402128    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402133    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402140    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402146    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402151    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402155    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402160    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402165    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402169    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402173    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402177    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402181    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402186    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.402197    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403441    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403452    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:14:26.405374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403457    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403461    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403468    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403475    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403479    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403483    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403487    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403491    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403495    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403499    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403504    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403508    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403513    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403517    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403521    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403525    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403529    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403533    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403537    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403541    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:14:26.406260 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403546    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403551    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403555    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403559    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403563    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403567    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403571    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403575    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403579    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403583    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403588    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403592    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403599    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403605    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403610    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403615    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403623    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403628    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403635    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:14:26.406943 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403640    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403645    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403649    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403654    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403658    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403662    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403666    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403671    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403675    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403679    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403682    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403686    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403690    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403694    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403698    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403702    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403706    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403710    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403714    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403718    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:14:26.407444 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403723    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403727    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403731    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403735    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403740    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403745    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403750    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403754    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403759    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403763    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403768    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403772    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403776    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403780    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403784    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403790    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403794    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403798    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403803    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403807    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:14:26.408110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403811    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403815    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403819    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403823    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403827    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403831    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403835    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403839    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403844    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403848    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403852    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403856    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403859    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403863    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403868    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403873    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403877    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.403881    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404551    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404562    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:14:26.408848 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404567    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404571    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404580    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404584    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404589    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404593    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404597    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404602    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404607    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404611    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404615    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404619    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404623    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404627    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404631    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404635    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404639    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404644    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404648    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404652    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:14:26.409354 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404656    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404660    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404664    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404668    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404672    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404677    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404681    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404685    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404689    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404694    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404698    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404703    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404707    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404711    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404715    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404720    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404725    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404729    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404733    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:14:26.409881 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404740    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404746    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404750    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404754    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404762    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404767    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404773    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404778    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404783    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404787    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404791    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404795    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404800    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404804    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404809    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404813    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404818    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404825    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404830    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:14:26.410450 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404835    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404840    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404844    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404848    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404853    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404857    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404862    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404866    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404870    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404874    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404879    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404883    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404887    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404891    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404895    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404899    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404903    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404908    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404912    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404916    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:14:26.410916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404920    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404924    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404928    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404932    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404938    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404942    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404947    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404951    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404955    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404959    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404963    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404967    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404972    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404976    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404980    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404984    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404989    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404993    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.404997    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405110    2584 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405123    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 20:14:26.411432 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405135    2584 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405143    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405151    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405156    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405163    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405169    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405174    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405179    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405185    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405191    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405195    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405200    2584 flags.go:64] FLAG: --cgroup-root=""
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405205    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405209    2584 flags.go:64] FLAG: --client-ca-file=""
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405213    2584 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405218    2584 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405225    2584 flags.go:64] FLAG: --cluster-domain=""
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405229    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405234    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405239    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405244    2584 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405251    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405258    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 20:14:26.411981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405264    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405270    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405275    2584 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405280    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405285    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405289    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405294    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405300    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405306    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405311    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405315    2584 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405320    2584 flags.go:64] FLAG: --enable-server="true"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405326    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405333    2584 flags.go:64] FLAG: --event-burst="100"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405338    2584 flags.go:64] FLAG: --event-qps="50"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405342    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405347    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405352    2584 flags.go:64] FLAG: --eviction-hard=""
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405358    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405362    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405367    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405372    2584 flags.go:64] FLAG: --eviction-soft=""
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405377    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405381    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405385    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 20:14:26.412542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405409    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405414    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405419    2584 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405423    2584 flags.go:64] FLAG: --feature-gates=""
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405430    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405434    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405441    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405446    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405451    2584 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405456    2584 flags.go:64] FLAG: --help="false"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405461    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405466    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405471    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405476    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405482    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405488    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405493    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405497    2584 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405502    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405507    2584 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405512    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405518    2584 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405523    2584 flags.go:64] FLAG: --kube-reserved=""
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405528    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 20:14:26.413216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405532    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405537    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405541    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405546    2584 flags.go:64] FLAG: --lock-file=""
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405550    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405555    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405560    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405569    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405573    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405578    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405583    2584 flags.go:64] FLAG: --logging-format="text"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405587    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405592    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405598    2584 flags.go:64] FLAG: --manifest-url=""
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405604    2584 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405611    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405616    2584 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405623    2584 flags.go:64] FLAG: --max-pods="110"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405628    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405633    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405637    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405642    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405647    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405652    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405657    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 20:14:26.413862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405668    2584 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405673    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405678    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405683    2584 flags.go:64] FLAG: --pod-cidr=""
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405688    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405698    2584 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405702    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405707    2584 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405712    2584 flags.go:64] FLAG: --port="10250"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405716    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405721    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0636a69882ea7bf2d"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405726    2584 flags.go:64] FLAG: --qos-reserved=""
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405731    2584 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405736    2584 flags.go:64] FLAG: --register-node="true"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405741    2584 flags.go:64] FLAG: --register-with-taints=""
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405747    2584 flags.go:64] FLAG: --registry-burst="10"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405751    2584 flags.go:64] FLAG: --registry-qps="5"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405756    2584 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405760    2584 flags.go:64] FLAG: --reserved-memory=""
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405766    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405771    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405776    2584 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405783    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405787    2584 flags.go:64] FLAG: --runonce="false"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405792    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 20:14:26.414505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405798    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405803    2584 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405808    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405813    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405817    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405822    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405827    2584 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405832    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405836    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405841    2584 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405846    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405851    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405857    2584 flags.go:64] FLAG: --system-cgroups=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405862    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405870    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405875    2584 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405879    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405886    2584 flags.go:64] FLAG: --tls-min-version=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405891    2584 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405895    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405900    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405905    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405909    2584 flags.go:64] FLAG: --v="2"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405916    2584 flags.go:64] FLAG: --version="false"
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405922    2584 flags.go:64] FLAG: --vmodule=""
Jul 13 20:14:26.415135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405928    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.405933    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406132    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406139    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406146    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406151    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406157    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406169    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406176    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406182    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406186    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406200    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406205    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406209    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406213    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406217    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406221    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406225    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406229    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406233    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:14:26.415802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406238    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406243    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406247    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406251    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406255    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406259    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406263    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406267    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406271    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406275    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406279    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406284    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406288    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406292    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406296    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406300    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406308    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406312    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406316    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:14:26.416308 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406322    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406326    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406331    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406335    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406339    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406344    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406351    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406356    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406361    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406365    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406369    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406374    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406378    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406382    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406388    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406409    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406414    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406418    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406422    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406426    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:14:26.416896 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406430    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406434    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406438    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406442    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406446    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406450    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406454    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406458    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406462    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406468    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406472    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406476    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406481    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406486    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406492    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406498    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406503    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406507    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406512    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:14:26.417374 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406516    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406520    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406524    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406528    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406532    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406537    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406541    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406545    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406550    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406554    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406558    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406562    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406566    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406570    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406575    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406579    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406583    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406587    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406592    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406596    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:14:26.417859 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406600    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406604    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.406610    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.407499    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.414762    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.414778    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.415748    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.415757    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415823    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415829    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415832    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415836    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415839    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415841    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415844    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:14:26.418356 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415847    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415850    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415852    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415856    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415858    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415861    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415864    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415866    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415869    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415873    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415877    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415879    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415882    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415885    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415888    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415890    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415893    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415896    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415900    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:14:26.418750 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415904    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415906    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415909    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415915    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415918    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415921    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415924    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415926    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415929    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415931    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415934    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415936    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415939    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415942    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415944    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415946    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415949    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415952    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415955    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415957    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:14:26.419201 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415959    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415962    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415964    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415967    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415969    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415972    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415974    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415977    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415979    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415982    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415984    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415987    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415989    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415992    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415994    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.415997    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416000    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416004    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416008    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:14:26.419765 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416011    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416013    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416016    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416018    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416020    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416023    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416025    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416028    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416030    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416032    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416035    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416037    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416040    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416042    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416045    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416048    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416050    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416053    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416055    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416057    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:14:26.420227 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416060    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416062    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416065    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416067    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416070    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416072    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416074    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416077    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416079    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416082    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416085    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416087    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416090    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416092    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.416098    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:14:26.420733 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416217    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416222    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416225    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416228    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416230    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416233    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416235    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416238    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416240    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416243    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416245    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416247    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416250    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416252    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416255    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416258    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416262    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416265    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416267    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:14:26.421100 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416270    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416273    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416275    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416277    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416280    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416282    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416284    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416287    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416289    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416292    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416295    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416297    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416300    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416302    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416306    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416309    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416312    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416315    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416318    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416321    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:14:26.421809 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416324    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416327    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416330    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416332    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416335    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416337    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416340    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416342    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416345    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416349    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416351    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416354    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416356    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416359    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416362    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416364    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416367    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416369    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416372    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:14:26.422599 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416374    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416377    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416379    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416382    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416385    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416387    2584 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416390    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416409    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416412    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416414    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416416    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416419    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416421    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416424    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416427    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416429    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416432    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416434    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416437    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416439    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:14:26.423171 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416442    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416444    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416447    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416449    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416451    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416454    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416456    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416459    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416462    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416464    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416466    2584 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416469    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416471    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416474    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416476    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416478    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416482    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416484    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416486    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416489    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:14:26.423740 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:26.416491    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:14:26.424258 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.416496    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:14:26.424258 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.416666    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 20:14:26.424258 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.418683    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 20:14:26.424258 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.420042    2584 server.go:1019] "Starting client certificate rotation"
Jul 13 20:14:26.424258 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.420158    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 20:14:26.424258 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.420238    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 20:14:26.424758 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.424740    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 20:14:26.425894 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.425877    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 20:14:26.434592 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.434574    2584 log.go:25] "Validated CRI v1 runtime API"
Jul 13 20:14:26.437387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.437370    2584 log.go:25] "Validated CRI v1 image API"
Jul 13 20:14:26.438214 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.438197    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 20:14:26.441579 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.441559    2584 fs.go:135] Filesystem UUIDs: map[56d5611c-4ea0-4cef-b061-fb16cbdbc5a4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b1198779-264b-4cf1-9eb7-646b5564ae98:/dev/nvme0n1p3]
Jul 13 20:14:26.441641 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.441578    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 20:14:26.447975 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.447852    2584 manager.go:216] Machine: {Timestamp:2026-07-13 20:14:26.446355732 +0000 UTC m=+0.347058384 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096068 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b5d709fab9bef0aa1f5c1faa4e3b8 SystemUUID:ec2b5d70-9fab-9bef-0aa1-f5c1faa4e3b8 BootID:ee5c74ae-df57-4e6c-8429-ed36539ed888 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:21:fe:2f:39:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:21:fe:2f:39:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:a0:04:32:69:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 20:14:26.447975 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.447969    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 20:14:26.448099 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.448061    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 20:14:26.450823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.450789    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 20:14:26.450986 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.450827    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 20:14:26.451642 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.451632    2584 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 20:14:26.451685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.451646    2584 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 20:14:26.451685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.451654    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 20:14:26.452313 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.452302    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 20:14:26.453700 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.453688    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 20:14:26.456659 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.456647    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 20:14:26.456795 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.456786    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 20:14:26.457388 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457378    2584 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 20:14:26.457439 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457413    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 20:14:26.457468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457439    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 20:14:26.457468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457452    2584 kubelet.go:398] "Adding apiserver pod source"
Jul 13 20:14:26.457468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457463    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 20:14:26.457949 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457927    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 20:14:26.458003 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.457952    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 20:14:26.460462 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.460444    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 20:14:26.464694 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.464412    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 20:14:26.465034 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465017    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 20:14:26.465091 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465072    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 20:14:26.465327 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465312    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 20:14:26.465409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465335    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 20:14:26.465409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465346    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 20:14:26.465409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465363    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 20:14:26.465409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465375    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 20:14:26.465409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465384    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 20:14:26.465929 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465908    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 20:14:26.465988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465932    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 20:14:26.465988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465947    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 20:14:26.465988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465956    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 20:14:26.465988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465969    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 20:14:26.465988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.465983    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 20:14:26.466880 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.466864    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 20:14:26.466946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.466896    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 20:14:26.470847 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.470832    2584 server.go:1268] "Started kubelet"
Jul 13 20:14:26.470975 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.470947    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 20:14:26.470975 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.470934    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 20:14:26.471078 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.471007    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 20:14:26.471837 ip-10-0-128-62 systemd[1]: Started Kubernetes Kubelet.
Jul 13 20:14:26.472203 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.472078    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 20:14:26.472423 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.472385    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 20:14:26.472481 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.472386    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 20:14:26.472593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.472536    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 20:14:26.473370 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.473358    2584 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 20:14:26.477546 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.477527    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 20:14:26.478547 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.478528    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 20:14:26.478634 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.478598    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 20:14:26.478634 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.478591    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:26.479348 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479187    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 20:14:26.479348 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479350    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 20:14:26.479516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479336    2584 factory.go:55] Registering systemd factory
Jul 13 20:14:26.479516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479246    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 20:14:26.479516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479414    2584 factory.go:223] Registration of the systemd container factory successfully
Jul 13 20:14:26.479516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479476    2584 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 20:14:26.479516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479482    2584 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 20:14:26.479903 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479664    2584 factory.go:153] Registering CRI-O factory
Jul 13 20:14:26.479903 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479678    2584 factory.go:223] Registration of the crio container factory successfully
Jul 13 20:14:26.479903 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479724    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 20:14:26.479903 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479750    2584 factory.go:103] Registering Raw factory
Jul 13 20:14:26.479903 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.479762    2584 manager.go:1196] Started watching for new ooms in manager
Jul 13 20:14:26.480131 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.480117    2584 manager.go:349] Starting recovery of all containers
Jul 13 20:14:26.481913 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.481860    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 20:14:26.482009 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.481952    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 20:14:26.483846 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.481898    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f29906a4890f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.470791439 +0000 UTC m=+0.371494094,LastTimestamp:2026-07-13 20:14:26.470791439 +0000 UTC m=+0.371494094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.490352 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.490213    2584 manager.go:354] Recovery completed
Jul 13 20:14:26.492497 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.492450    2584 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 13 20:14:26.496286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.496271    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.498024 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498008    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.498098 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498039    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.498098 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498049    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.498651 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498636    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 20:14:26.498651 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498650    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 20:14:26.498755 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498667    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.498755 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.498669    2584 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 20:14:26.499348 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.499334    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.499387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.499363    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.499387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.499375    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.499993 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.499982    2584 policy_none.go:49] "None policy: Start"
Jul 13 20:14:26.500038 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.499997    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 20:14:26.500038 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.500007    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 20:14:26.500885 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.500823    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f2990844141d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-62.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.498024477 +0000 UTC m=+0.398727130,LastTimestamp:2026-07-13 20:14:26.498024477 +0000 UTC m=+0.398727130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.501213 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.501201    2584 policy_none.go:47] "Start"
Jul 13 20:14:26.508862 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.508792    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f299084460df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-62.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.498044127 +0000 UTC m=+0.398746780,LastTimestamp:2026-07-13 20:14:26.498044127 +0000 UTC m=+0.398746780,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.520133 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.520050    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f299084484e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-62.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.498053347 +0000 UTC m=+0.398756000,LastTimestamp:2026-07-13 20:14:26.498053347 +0000 UTC m=+0.398756000,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.531628 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.531543    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-62.ec2.internal.18c1f2990844141d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f2990844141d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-62.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.498024477 +0000 UTC m=+0.398727130,LastTimestamp:2026-07-13 20:14:26.499348442 +0000 UTC m=+0.400051094,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.541033 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.540942    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-62.ec2.internal.18c1f299084460df\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f299084460df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-62.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.498044127 +0000 UTC m=+0.398746780,LastTimestamp:2026-07-13 20:14:26.499369488 +0000 UTC m=+0.400072141,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.541141 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541129    2584 manager.go:337] "Starting Device Plugin manager"
Jul 13 20:14:26.541211 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.541161    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 20:14:26.541211 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541170    2584 server.go:87] "Starting device plugin registration server"
Jul 13 20:14:26.541500 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541486    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 20:14:26.541590 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541499    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 20:14:26.541640 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541626    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 20:14:26.541741 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541720    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 20:14:26.541741 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.541731    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 20:14:26.542336 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.542313    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 20:14:26.542428 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.542367    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:26.553092 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.553012    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-62.ec2.internal.18c1f299084484e3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f299084484e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-62.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.498053347 +0000 UTC m=+0.398756000,LastTimestamp:2026-07-13 20:14:26.499378841 +0000 UTC m=+0.400081495,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.561652 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.561573    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-62.ec2.internal.18c1f2990afa6387  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-62.ec2.internal,UID:ip-10-0-128-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-62.ec2.internal,},FirstTimestamp:2026-07-13 20:14:26.543526791 +0000 UTC m=+0.444229432,LastTimestamp:2026-07-13 20:14:26.543526791 +0000 UTC m=+0.444229432,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-62.ec2.internal,}"
Jul 13 20:14:26.602216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.602179    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-47g8h"
Jul 13 20:14:26.610353 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.610329    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-47g8h"
Jul 13 20:14:26.630221 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.630174    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 20:14:26.631524 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.631501    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 20:14:26.631638 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.631539    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 20:14:26.631638 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.631562    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 20:14:26.631638 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.631593    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 20:14:26.642252 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.642230    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.643241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.643219    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.643369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.643251    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.643369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.643261    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.643369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.643285    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.644615 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.644596    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:14:26.650574 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.650524    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.650574 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.650548    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-62.ec2.internal\": node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:26.662287 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.662267    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:26.733442 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.732554    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"]
Jul 13 20:14:26.733601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.733488    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.734485 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.734471    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.734553 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.734504    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.734553 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.734520    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.735969 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.735956    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.736098 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.736084    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.736135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.736113    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.736706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.736688    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.736786 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.736723    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.736786 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.736739    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.737243 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.737227    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.737292 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.737254    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.737292 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.737268    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.737946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.737933    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.737992 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.737958    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:26.738613 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.738598    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:26.738701 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.738624    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:26.738701 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.738634    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:26.760096 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.760068    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-62.ec2.internal\" not found" node="ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.762676 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.762657    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:26.764615 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.764599    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-62.ec2.internal\" not found" node="ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.781271 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.781238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/158b1adfd8b8e1f2922fb33d65d314e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal\" (UID: \"158b1adfd8b8e1f2922fb33d65d314e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.781271 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.781272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/158b1adfd8b8e1f2922fb33d65d314e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal\" (UID: \"158b1adfd8b8e1f2922fb33d65d314e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.781446 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.781289    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/121fbd70dde46c6dca67ebc08b171187-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.ec2.internal\" (UID: \"121fbd70dde46c6dca67ebc08b171187\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.863060 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.863033    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:26.882277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.882188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/158b1adfd8b8e1f2922fb33d65d314e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal\" (UID: \"158b1adfd8b8e1f2922fb33d65d314e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.882332 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.882301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/158b1adfd8b8e1f2922fb33d65d314e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal\" (UID: \"158b1adfd8b8e1f2922fb33d65d314e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.882332 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.882322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/121fbd70dde46c6dca67ebc08b171187-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.ec2.internal\" (UID: \"121fbd70dde46c6dca67ebc08b171187\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.882409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.882235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/158b1adfd8b8e1f2922fb33d65d314e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal\" (UID: \"158b1adfd8b8e1f2922fb33d65d314e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.882409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.882358    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/121fbd70dde46c6dca67ebc08b171187-config\") pod \"kube-apiserver-proxy-ip-10-0-128-62.ec2.internal\" (UID: \"121fbd70dde46c6dca67ebc08b171187\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.882479 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:26.882429    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/158b1adfd8b8e1f2922fb33d65d314e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal\" (UID: \"158b1adfd8b8e1f2922fb33d65d314e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:26.964208 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:26.964140    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.061780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.061751    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:27.064383 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.064368    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.067673 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.067659    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:27.168370 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.168321    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.272021 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.271949    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.372190 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.372144    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.420112 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.420073    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 20:14:27.420774 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.420221    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 20:14:27.473325 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.473272    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.564722 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.564613    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:14:27.565726 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.565699    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:14:27.565867 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.565747    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:14:27.565867 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.565765    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:14:27.573654 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.573636    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.615383 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.615340    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 20:09:26 +0000 UTC" deadline="2028-04-19 12:47:00.87567294 +0000 UTC"
Jul 13 20:14:27.615383 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.615379    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15496h32m33.26029722s"
Jul 13 20:14:27.650890 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.650862    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:14:27.674070 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:27.674042    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-62.ec2.internal\" not found"
Jul 13 20:14:27.705238 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.705217    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:14:27.779855 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.779836    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:27.791052 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:27.791017    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158b1adfd8b8e1f2922fb33d65d314e7.slice/crio-1291da92ca12d0cb0222da22b17c147042ac0add0cde5debd872cda497ec0c9f WatchSource:0}: Error finding container 1291da92ca12d0cb0222da22b17c147042ac0add0cde5debd872cda497ec0c9f: Status 404 returned error can't find the container with id 1291da92ca12d0cb0222da22b17c147042ac0add0cde5debd872cda497ec0c9f
Jul 13 20:14:27.794420 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.794319    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 20:14:27.797802 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:27.797774    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121fbd70dde46c6dca67ebc08b171187.slice/crio-2f26b9aeea9e0ae56181f0a24269f0488094ad52cbcab9ec37ce4e29a26cefc1 WatchSource:0}: Error finding container 2f26b9aeea9e0ae56181f0a24269f0488094ad52cbcab9ec37ce4e29a26cefc1: Status 404 returned error can't find the container with id 2f26b9aeea9e0ae56181f0a24269f0488094ad52cbcab9ec37ce4e29a26cefc1
Jul 13 20:14:27.798395 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.798371    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:14:27.799860 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.799722    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal"
Jul 13 20:14:27.812834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.812814    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 20:14:27.830600 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:27.830530    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:14:28.459091 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.459048    2584 apiserver.go:52] "Watching apiserver"
Jul 13 20:14:28.468526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.468458    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 20:14:28.469644 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.469617    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-htbsc","openshift-network-diagnostics/network-check-target-4tvtl","openshift-network-operator/iptables-alerter-4vzzb","openshift-ovn-kubernetes/ovnkube-node-cv2hn","kube-system/global-pull-secret-syncer-gvtll","kube-system/konnectivity-agent-6s44f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns","openshift-cluster-node-tuning-operator/tuned-2nn7d","openshift-dns/node-resolver-4nkxs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-wdwzm","openshift-multus/network-metrics-daemon-2vxjq","kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal","openshift-image-registry/node-ca-4h87g"]
Jul 13 20:14:28.471340 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.471318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.472657 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.472632    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.473856 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.473835    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.474862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.474842    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 20:14:28.475209 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.475191    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.475318 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.475216    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-prtlx\""
Jul 13 20:14:28.475382 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.475339    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.475523 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.475506    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.475792 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.475774    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 20:14:28.475872 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.475514    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.476863 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.476847    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.476961 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.476934    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:28.477213 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.477194    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 20:14:28.477674 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.477649    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6nrlx\""
Jul 13 20:14:28.477780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.477713    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.478389 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.478372    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 20:14:28.478854 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.478833    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.479721 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.479701    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.479856 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.479840    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.479930 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.479916    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 20:14:28.480040 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.480022    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 20:14:28.480150 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.480118    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.481764 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.481745    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.481876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.481778    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vpjls\""
Jul 13 20:14:28.481876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.481856    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 20:14:28.482093 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.482079    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.482764 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.482731    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 20:14:28.482861 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.482779    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 20:14:28.482923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.482910    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 20:14:28.483144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.483128    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.484065 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.484046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:28.484150 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.484112    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:28.485487 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.485470    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.486876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.486859    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.488099 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.488069    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:28.488222 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.488140    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:28.489669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.489652    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.489820 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.489799    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.489893 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.489822    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 20:14:28.490220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.489971    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 20:14:28.490220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.490078    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 20:14:28.490220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.490107    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w77b6\""
Jul 13 20:14:28.490220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.490125    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.491402 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkz5\" (UniqueName: \"kubernetes.io/projected/45a5d5fc-dc3a-4d65-b776-21e0697f615d-kube-api-access-jgkz5\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.491483 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491453    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-run-netns\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491531 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491501    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-log-socket\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-socket-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.491604 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.491604 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491594    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-etc-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491672 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-ovn\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491672 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-cni-bin\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491781 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491686    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-cni-netd\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491781 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491722    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88f467e-b9f0-403b-a2f4-f08a7980c965-dbus\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.491781 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491750    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-os-release\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.491879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491780    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-ovnkube-script-lib\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491814    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzmv\" (UniqueName: \"kubernetes.io/projected/dc4ac74e-a556-44b9-b935-0136456b2898-kube-api-access-hvzmv\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.491879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491845    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-host-slash\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.491879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491875    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-node-log\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491996 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491996 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-systemd\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.491996 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492085 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.491999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492085 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492034    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-tmp-dir\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.492085 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492071    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-sys-fs\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.492180 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492099    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.492180 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492131    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd8e839d-4afb-4c79-b577-6a8a6b3170ca-konnectivity-ca\") pod \"konnectivity-agent-6s44f\" (UID: \"fd8e839d-4afb-4c79-b577-6a8a6b3170ca\") " pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.492180 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492164    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.492267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-kubelet\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-slash\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e75521d-6024-430e-acc2-4ea6407907e7-ovn-node-metrics-cert\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492365 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492284    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-hosts-file\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.492365 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492322    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-iptables-alerter-script\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.492365 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492354    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88f467e-b9f0-403b-a2f4-f08a7980c965-kubelet-config\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.492464 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492380    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd8e839d-4afb-4c79-b577-6a8a6b3170ca-agent-certs\") pod \"konnectivity-agent-6s44f\" (UID: \"fd8e839d-4afb-4c79-b577-6a8a6b3170ca\") " pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.492464 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492413    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.492464 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-ovnkube-config\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492579 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492493    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-env-overrides\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492579 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.492579 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492566    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5g6v\" (UniqueName: \"kubernetes.io/projected/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-kube-api-access-d5g6v\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.492669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492598    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-system-cni-dir\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.492669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492625    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-systemd-units\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492729 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492682    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-var-lib-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492729 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlpjt\" (UniqueName: \"kubernetes.io/projected/9e75521d-6024-430e-acc2-4ea6407907e7-kube-api-access-vlpjt\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.492792 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492746    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24w4\" (UniqueName: \"kubernetes.io/projected/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-kube-api-access-l24w4\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.492792 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492776    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-registration-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.492853 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-device-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.492853 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492834    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cnibin\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.492917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492866    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.492917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.492899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.493845 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.493688    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 20:14:28.493845 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.493723    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.494032 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.494011    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 20:14:28.494849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.494361    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n9v4r\""
Jul 13 20:14:28.494849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.494427    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-68595\""
Jul 13 20:14:28.494849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.494455    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hcqn\""
Jul 13 20:14:28.494849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.494766    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 20:14:28.495099 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.494992    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5tk4c\""
Jul 13 20:14:28.495737 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.495203    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-99rpl\""
Jul 13 20:14:28.504154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.504132    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 20:14:28.536956 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.536931    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nhsgf"
Jul 13 20:14:28.557518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.557480    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nhsgf"
Jul 13 20:14:28.581434 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.581395    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 20:14:28.593622 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593590    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-os-release\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.593783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593626    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-ovnkube-script-lib\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.593783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-os-release\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.593783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593776    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzmv\" (UniqueName: \"kubernetes.io/projected/dc4ac74e-a556-44b9-b935-0136456b2898-kube-api-access-hvzmv\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593808    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-node-log\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593838    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lw4r\" (UniqueName: \"kubernetes.io/projected/2efc0e75-f975-4046-b42f-c142d7ed511b-kube-api-access-7lw4r\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593867    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysconfig\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593917    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-systemd\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593944    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2efc0e75-f975-4046-b42f-c142d7ed511b-serviceca\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593970    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-netns\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.593925    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-node-log\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.594005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594000    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd8e839d-4afb-4c79-b577-6a8a6b3170ca-konnectivity-ca\") pod \"konnectivity-agent-6s44f\" (UID: \"fd8e839d-4afb-4c79-b577-6a8a6b3170ca\") " pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594092    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-kubelet\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-hosts-file\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.594234    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594286    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-ovnkube-script-lib\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.594311    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:29.094289845 +0000 UTC m=+2.994992508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594334    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-hosts-file\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-kubelet\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.594422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594364    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-cni-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594507    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-conf-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-iptables-alerter-script\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594630    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88f467e-b9f0-403b-a2f4-f08a7980c965-kubelet-config\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594661    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88f467e-b9f0-403b-a2f4-f08a7980c965-kubelet-config\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd8e839d-4afb-4c79-b577-6a8a6b3170ca-agent-certs\") pod \"konnectivity-agent-6s44f\" (UID: \"fd8e839d-4afb-4c79-b577-6a8a6b3170ca\") " pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-env-overrides\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-run\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594810    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-host\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594835    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-os-release\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594860    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5g6v\" (UniqueName: \"kubernetes.io/projected/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-kube-api-access-d5g6v\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594920    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-system-cni-dir\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594936    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fd8e839d-4afb-4c79-b577-6a8a6b3170ca-konnectivity-ca\") pod \"konnectivity-agent-6s44f\" (UID: \"fd8e839d-4afb-4c79-b577-6a8a6b3170ca\") " pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlpjt\" (UniqueName: \"kubernetes.io/projected/9e75521d-6024-430e-acc2-4ea6407907e7-kube-api-access-vlpjt\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594946    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.595028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.594974    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-multus-certs\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595003    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-system-cni-dir\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595046    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544tj\" (UniqueName: \"kubernetes.io/projected/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-kube-api-access-544tj\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595081    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-registration-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-registration-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-device-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595119    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595180    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595188    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-device-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595226    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595237    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-iptables-alerter-script\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595250    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkz5\" (UniqueName: \"kubernetes.io/projected/45a5d5fc-dc3a-4d65-b776-21e0697f615d-kube-api-access-jgkz5\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595229    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-env-overrides\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595268    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-run-netns\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595291    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-cni-netd\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595320    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-cni-bin\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595341    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-run-netns\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.595852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595347    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-socket-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595373    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595393    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-etc-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-cni-netd\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595452    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-etc-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595455    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-ovn\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595491    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595496    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-ovn\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595514    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-socket-dir\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595542    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-k8s-cni-cncf-io\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595568    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98a95222-55d4-4845-a53a-d00cecd8f210-multus-daemon-config\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595587    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-host-slash\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-var-lib-kubelet\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595636    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-socket-dir-parent\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-systemd\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595660    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-host-slash\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.596601 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595669    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595674    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595708    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595738    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-run-systemd\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-run-ovn-kubernetes\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595740    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-tmp-dir\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595770    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-modprobe-d\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595766    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595789    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-sys\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595806    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-lib-modules\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98a95222-55d4-4845-a53a-d00cecd8f210-cni-binary-copy\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595837    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2efc0e75-f975-4046-b42f-c142d7ed511b-host\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-kubernetes\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-sys-fs\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-slash\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e75521d-6024-430e-acc2-4ea6407907e7-ovn-node-metrics-cert\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.597377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595922    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-sys-fs\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595930    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysctl-d\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysctl-conf\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-slash\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595959    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-hostroot\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595979    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.595996    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-ovnkube-config\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24w4\" (UniqueName: \"kubernetes.io/projected/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-kube-api-access-l24w4\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-tmp-dir\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596042    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9199a222-defd-4294-b6c6-86341b1bc881-etc-tuned\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596057    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-cnibin\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596073    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596093    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-systemd-units\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-var-lib-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-kubelet\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cnibin\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596155    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-log-socket\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596171    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-cni-bin\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596186    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9199a222-defd-4294-b6c6-86341b1bc881-tmp\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-system-cni-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596218    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-cni-multus\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zf5d\" (UniqueName: \"kubernetes.io/projected/98a95222-55d4-4845-a53a-d00cecd8f210-kube-api-access-9zf5d\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596249    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596264    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6mg\" (UniqueName: \"kubernetes.io/projected/9199a222-defd-4294-b6c6-86341b1bc881-kube-api-access-ck6mg\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596279    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-etc-kubernetes\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596294    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88f467e-b9f0-403b-a2f4-f08a7980c965-dbus\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-log-socket\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88f467e-b9f0-403b-a2f4-f08a7980c965-dbus\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596443    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-host-cni-bin\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cnibin\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596588    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-systemd-units\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596577    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e75521d-6024-430e-acc2-4ea6407907e7-var-lib-openvswitch\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.596600    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc4ac74e-a556-44b9-b935-0136456b2898-etc-selinux\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.597162    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e75521d-6024-430e-acc2-4ea6407907e7-ovnkube-config\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.598966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.597929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/45a5d5fc-dc3a-4d65-b776-21e0697f615d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.599621 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.599519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e75521d-6024-430e-acc2-4ea6407907e7-ovn-node-metrics-cert\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.599662 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.599640    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fd8e839d-4afb-4c79-b577-6a8a6b3170ca-agent-certs\") pod \"konnectivity-agent-6s44f\" (UID: \"fd8e839d-4afb-4c79-b577-6a8a6b3170ca\") " pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.604911 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.604882    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzmv\" (UniqueName: \"kubernetes.io/projected/dc4ac74e-a556-44b9-b935-0136456b2898-kube-api-access-hvzmv\") pod \"aws-ebs-csi-driver-node-gpmns\" (UID: \"dc4ac74e-a556-44b9-b935-0136456b2898\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.605514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.605492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlpjt\" (UniqueName: \"kubernetes.io/projected/9e75521d-6024-430e-acc2-4ea6407907e7-kube-api-access-vlpjt\") pod \"ovnkube-node-cv2hn\" (UID: \"9e75521d-6024-430e-acc2-4ea6407907e7\") " pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.606872 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.606849    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkz5\" (UniqueName: \"kubernetes.io/projected/45a5d5fc-dc3a-4d65-b776-21e0697f615d-kube-api-access-jgkz5\") pod \"multus-additional-cni-plugins-wdwzm\" (UID: \"45a5d5fc-dc3a-4d65-b776-21e0697f615d\") " pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.607031 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.607012    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24w4\" (UniqueName: \"kubernetes.io/projected/33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34-kube-api-access-l24w4\") pod \"node-resolver-4nkxs\" (UID: \"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34\") " pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.607403 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.607385    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5g6v\" (UniqueName: \"kubernetes.io/projected/a5d6166d-6f7c-4637-b1eb-42687ded1a6b-kube-api-access-d5g6v\") pod \"iptables-alerter-4vzzb\" (UID: \"a5d6166d-6f7c-4637-b1eb-42687ded1a6b\") " pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.637969 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.637907    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal" event={"ID":"121fbd70dde46c6dca67ebc08b171187","Type":"ContainerStarted","Data":"2f26b9aeea9e0ae56181f0a24269f0488094ad52cbcab9ec37ce4e29a26cefc1"}
Jul 13 20:14:28.639451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.639416    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal" event={"ID":"158b1adfd8b8e1f2922fb33d65d314e7","Type":"ContainerStarted","Data":"1291da92ca12d0cb0222da22b17c147042ac0add0cde5debd872cda497ec0c9f"}
Jul 13 20:14:28.696956 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.696924    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544tj\" (UniqueName: \"kubernetes.io/projected/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-kube-api-access-544tj\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.696973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-cni-bin\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.696991    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-k8s-cni-cncf-io\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98a95222-55d4-4845-a53a-d00cecd8f210-multus-daemon-config\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697047    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-var-lib-kubelet\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-socket-dir-parent\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-modprobe-d\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-k8s-cni-cncf-io\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697207    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-var-lib-kubelet\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-cni-bin\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697276    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-sys\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-socket-dir-parent\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697308    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-lib-modules\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697280    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-modprobe-d\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-sys\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697355    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98a95222-55d4-4845-a53a-d00cecd8f210-cni-binary-copy\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697388    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-lib-modules\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2efc0e75-f975-4046-b42f-c142d7ed511b-host\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697419    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-kubernetes\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697421    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2efc0e75-f975-4046-b42f-c142d7ed511b-host\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-kubernetes\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697491    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysctl-d\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697514    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysctl-conf\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-hostroot\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.697593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697558    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9199a222-defd-4294-b6c6-86341b1bc881-etc-tuned\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697638    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysctl-d\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-cnibin\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.697951    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysctl-conf\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698115    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/98a95222-55d4-4845-a53a-d00cecd8f210-multus-daemon-config\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698123    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-kubelet\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698169    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9199a222-defd-4294-b6c6-86341b1bc881-tmp\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.698256    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698316    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-kubelet\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.698332    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:29.19831195 +0000 UTC m=+3.099014592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:28.698387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698366    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-system-cni-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/98a95222-55d4-4845-a53a-d00cecd8f210-cni-binary-copy\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698429    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-cni-multus\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-var-lib-cni-multus\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zf5d\" (UniqueName: \"kubernetes.io/projected/98a95222-55d4-4845-a53a-d00cecd8f210-kube-api-access-9zf5d\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6mg\" (UniqueName: \"kubernetes.io/projected/9199a222-defd-4294-b6c6-86341b1bc881-kube-api-access-ck6mg\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698600    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-etc-kubernetes\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698602    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-hostroot\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698566    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-system-cni-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698673    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lw4r\" (UniqueName: \"kubernetes.io/projected/2efc0e75-f975-4046-b42f-c142d7ed511b-kube-api-access-7lw4r\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-cnibin\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysconfig\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-systemd\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.698978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2efc0e75-f975-4046-b42f-c142d7ed511b-serviceca\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.698999    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-netns\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699055    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-cni-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-conf-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-run\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-host\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699208    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-os-release\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-multus-certs\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-multus-certs\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-sysconfig\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699452    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-etc-kubernetes\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-conf-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699624    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-host-run-netns\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.699733 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-multus-cni-dir\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.700357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699779    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-run\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.700357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-host\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.700357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699858    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/98a95222-55d4-4845-a53a-d00cecd8f210-os-release\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.700357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.699973    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9199a222-defd-4294-b6c6-86341b1bc881-etc-systemd\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.700357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.700169    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2efc0e75-f975-4046-b42f-c142d7ed511b-serviceca\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.702268 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.701969    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9199a222-defd-4294-b6c6-86341b1bc881-etc-tuned\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.703619 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.703595    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:14:28.703619 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.703622    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:14:28.703787 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.703638    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:28.703787 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:28.703707    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:14:29.20368877 +0000 UTC m=+3.104391427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:28.704360 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.704297    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9199a222-defd-4294-b6c6-86341b1bc881-tmp\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.706169 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.706149    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-544tj\" (UniqueName: \"kubernetes.io/projected/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-kube-api-access-544tj\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:28.709825 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.709768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zf5d\" (UniqueName: \"kubernetes.io/projected/98a95222-55d4-4845-a53a-d00cecd8f210-kube-api-access-9zf5d\") pod \"multus-htbsc\" (UID: \"98a95222-55d4-4845-a53a-d00cecd8f210\") " pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.711095 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.711065    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lw4r\" (UniqueName: \"kubernetes.io/projected/2efc0e75-f975-4046-b42f-c142d7ed511b-kube-api-access-7lw4r\") pod \"node-ca-4h87g\" (UID: \"2efc0e75-f975-4046-b42f-c142d7ed511b\") " pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.719650 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.719622    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6mg\" (UniqueName: \"kubernetes.io/projected/9199a222-defd-4294-b6c6-86341b1bc881-kube-api-access-ck6mg\") pod \"tuned-2nn7d\" (UID: \"9199a222-defd-4294-b6c6-86341b1bc881\") " pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.784840 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.784803    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns"
Jul 13 20:14:28.794831 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.794798    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wdwzm"
Jul 13 20:14:28.806154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.806119    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4vzzb"
Jul 13 20:14:28.812088 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.812054    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:28.819782 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.819756    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:28.827714 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.827692    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4nkxs"
Jul 13 20:14:28.835532 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.835508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4h87g"
Jul 13 20:14:28.843409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.843373    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nn7d"
Jul 13 20:14:28.850162 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.850143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-htbsc"
Jul 13 20:14:28.852874 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:28.852853    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:14:29.080235 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:29.080193    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d6166d_6f7c_4637_b1eb_42687ded1a6b.slice/crio-6e1da0d64889d72ed04a539c25e94b39480e705b86710fcc0103abd38bc00025 WatchSource:0}: Error finding container 6e1da0d64889d72ed04a539c25e94b39480e705b86710fcc0103abd38bc00025: Status 404 returned error can't find the container with id 6e1da0d64889d72ed04a539c25e94b39480e705b86710fcc0103abd38bc00025
Jul 13 20:14:29.086542 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:29.086518    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efc0e75_f975_4046_b42f_c142d7ed511b.slice/crio-dd215c3094e0526e4c6b0ae693c648e63c0b906ab9be852268b0416059a4f569 WatchSource:0}: Error finding container dd215c3094e0526e4c6b0ae693c648e63c0b906ab9be852268b0416059a4f569: Status 404 returned error can't find the container with id dd215c3094e0526e4c6b0ae693c648e63c0b906ab9be852268b0416059a4f569
Jul 13 20:14:29.086814 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:29.086787    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8e839d_4afb_4c79_b577_6a8a6b3170ca.slice/crio-60aba02e935d222e939075dbf16d61b874b95ad10b931aa803c942c2f1da3621 WatchSource:0}: Error finding container 60aba02e935d222e939075dbf16d61b874b95ad10b931aa803c942c2f1da3621: Status 404 returned error can't find the container with id 60aba02e935d222e939075dbf16d61b874b95ad10b931aa803c942c2f1da3621
Jul 13 20:14:29.088551 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:29.088397    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e75521d_6024_430e_acc2_4ea6407907e7.slice/crio-e53cecdb5bc2603a5fd218a7567788f96b2e699bbe2cc9170a7e8064937c2fa6 WatchSource:0}: Error finding container e53cecdb5bc2603a5fd218a7567788f96b2e699bbe2cc9170a7e8064937c2fa6: Status 404 returned error can't find the container with id e53cecdb5bc2603a5fd218a7567788f96b2e699bbe2cc9170a7e8064937c2fa6
Jul 13 20:14:29.090956 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:29.090326    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4ac74e_a556_44b9_b935_0136456b2898.slice/crio-c4a8b0ed00050d697071b69b95d6784a8e5bbee61c1e961740c7d24dda91dd51 WatchSource:0}: Error finding container c4a8b0ed00050d697071b69b95d6784a8e5bbee61c1e961740c7d24dda91dd51: Status 404 returned error can't find the container with id c4a8b0ed00050d697071b69b95d6784a8e5bbee61c1e961740c7d24dda91dd51
Jul 13 20:14:29.091673 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:14:29.091650    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bbf7c8_d5f1_4bfe_bbe4_75eaa3829d34.slice/crio-5d55d3e6709a6c0086f4fc8d86f44acbf9498ee971c10ed8963e82e94ae3c644 WatchSource:0}: Error finding container 5d55d3e6709a6c0086f4fc8d86f44acbf9498ee971c10ed8963e82e94ae3c644: Status 404 returned error can't find the container with id 5d55d3e6709a6c0086f4fc8d86f44acbf9498ee971c10ed8963e82e94ae3c644
Jul 13 20:14:29.102381 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.102359    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:29.102583 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.102547    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:29.102646 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.102612    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:30.102591414 +0000 UTC m=+4.003294054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:29.202950 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.202748    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:29.203127 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.202911    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:29.203127 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.203015    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:30.202999046 +0000 UTC m=+4.103701687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:29.303799 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.303767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:29.303953 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.303935    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:14:29.303996 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.303958    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:14:29.303996 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.303970    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:29.304062 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.304025    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:14:30.304011241 +0000 UTC m=+4.204713900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:29.560505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.560450    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:09:28 +0000 UTC" deadline="2028-01-13 16:02:59.545940623 +0000 UTC"
Jul 13 20:14:29.560505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.560501    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13171h48m29.985445689s"
Jul 13 20:14:29.632180 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.632143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:29.632365 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:29.632265    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:29.647213 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.647173    2584 generic.go:356] "Generic (PLEG): container finished" podID="158b1adfd8b8e1f2922fb33d65d314e7" containerID="045f9dc012b54e58a4341e99d0e9d3ee2884b447b89c24c8c127006f603bae38" exitCode=0
Jul 13 20:14:29.647905 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.647877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal" event={"ID":"158b1adfd8b8e1f2922fb33d65d314e7","Type":"ContainerDied","Data":"045f9dc012b54e58a4341e99d0e9d3ee2884b447b89c24c8c127006f603bae38"}
Jul 13 20:14:29.651787 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.651700    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-htbsc" event={"ID":"98a95222-55d4-4845-a53a-d00cecd8f210","Type":"ContainerStarted","Data":"02d1d7bddafee37084fb40d47d6705ae8a0fb8499b54016a5ee61c4ef5facb5e"}
Jul 13 20:14:29.654073 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.654005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4nkxs" event={"ID":"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34","Type":"ContainerStarted","Data":"5d55d3e6709a6c0086f4fc8d86f44acbf9498ee971c10ed8963e82e94ae3c644"}
Jul 13 20:14:29.657911 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.657849    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4h87g" event={"ID":"2efc0e75-f975-4046-b42f-c142d7ed511b","Type":"ContainerStarted","Data":"dd215c3094e0526e4c6b0ae693c648e63c0b906ab9be852268b0416059a4f569"}
Jul 13 20:14:29.661257 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.661198    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nn7d" event={"ID":"9199a222-defd-4294-b6c6-86341b1bc881","Type":"ContainerStarted","Data":"98e5b4b789eb50c4b77a20096d09907b7170d0acdfe9857e4fa5e8760dba89cb"}
Jul 13 20:14:29.663622 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.663550    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerStarted","Data":"14d3a72c2c9d482237757a13a6b2005f29ab95801ce9c411952739993e70770c"}
Jul 13 20:14:29.681256 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.681222    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns" event={"ID":"dc4ac74e-a556-44b9-b935-0136456b2898","Type":"ContainerStarted","Data":"c4a8b0ed00050d697071b69b95d6784a8e5bbee61c1e961740c7d24dda91dd51"}
Jul 13 20:14:29.692997 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.692859    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"e53cecdb5bc2603a5fd218a7567788f96b2e699bbe2cc9170a7e8064937c2fa6"}
Jul 13 20:14:29.720384 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.720072    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6s44f" event={"ID":"fd8e839d-4afb-4c79-b577-6a8a6b3170ca","Type":"ContainerStarted","Data":"60aba02e935d222e939075dbf16d61b874b95ad10b931aa803c942c2f1da3621"}
Jul 13 20:14:29.725516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:29.725435    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vzzb" event={"ID":"a5d6166d-6f7c-4637-b1eb-42687ded1a6b","Type":"ContainerStarted","Data":"6e1da0d64889d72ed04a539c25e94b39480e705b86710fcc0103abd38bc00025"}
Jul 13 20:14:30.112073 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.112022    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:30.112260 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.112247    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:30.112343 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.112317    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:32.112297831 +0000 UTC m=+6.013000481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:30.213657 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.213399    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:30.213657 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.213642    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:30.213900 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.213718    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:32.213695999 +0000 UTC m=+6.114398663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:30.314716 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.314574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:30.314981 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.314812    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:14:30.314981 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.314842    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:14:30.314981 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.314858    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:30.314981 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.314924    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:14:32.314902654 +0000 UTC m=+6.215605311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:30.561415 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.561256    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:09:28 +0000 UTC" deadline="2028-03-22 10:00:27.458787794 +0000 UTC"
Jul 13 20:14:30.561415 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.561327    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14821h45m56.897466061s"
Jul 13 20:14:30.634289 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.633619    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:30.634289 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.633765    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:30.634289 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.633819    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:30.634289 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:30.633970    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:30.750380 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.749658    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal" event={"ID":"158b1adfd8b8e1f2922fb33d65d314e7","Type":"ContainerStarted","Data":"bbc019f28e2aed9143e3c3e8ebb50f77dcf5b5c9303d67714858f849883e23e4"}
Jul 13 20:14:30.770131 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:30.769048    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-62.ec2.internal" podStartSLOduration=2.769025236 podStartE2EDuration="2.769025236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:30.768882852 +0000 UTC m=+4.669585521" watchObservedRunningTime="2026-07-13 20:14:30.769025236 +0000 UTC m=+4.669727903"
Jul 13 20:14:31.632785 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:31.632726    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:31.633499 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:31.632874    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:32.130431 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:32.130378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:32.130622 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.130581    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:32.130726 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.130641    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:36.130623681 +0000 UTC m=+10.031326327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:32.231780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:32.231742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:32.231978 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.231946    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:32.232037 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.232004    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:36.231985083 +0000 UTC m=+10.132687727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:32.332360 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:32.332325    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:32.332539 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.332495    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:14:32.332539 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.332517    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:14:32.332539 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.332530    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:32.332717 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.332616    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:14:36.332597474 +0000 UTC m=+10.233300122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:32.635657 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:32.634673    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:32.635657 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.634808    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:32.635657 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:32.635143    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:32.635657 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:32.635248    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:33.631937 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:33.631895    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:33.632578 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:33.632172    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:34.634422 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:34.634366    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:34.634855 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:34.634519    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:34.634855 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:34.634519    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:34.634855 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:34.634652    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:35.632851 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:35.632295    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:35.632851 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:35.632453    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:36.166949 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:36.166902    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:36.167482 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.167061    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:36.167482 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.167109    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:44.167097558 +0000 UTC m=+18.067800201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:36.268264 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:36.268223    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:36.268487 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.268470    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:36.268602 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.268547    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:14:44.268527823 +0000 UTC m=+18.169230465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:36.369462 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:36.369413    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:36.369651 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.369577    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:14:36.369651 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.369598    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:14:36.369651 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.369610    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:36.369801 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.369673    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:14:44.369655542 +0000 UTC m=+18.270358187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:36.633323 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:36.633286    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:36.633517 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:36.633297    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:36.633517 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.633420    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:36.633517 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:36.633477    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:37.631994 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:37.631958    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:37.632621 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:37.632093    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:38.632616 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:38.632584    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:38.632999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:38.632598    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:38.632999 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:38.632720    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:38.632999 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:38.632783    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:39.632346 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:39.632301    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:39.632546 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:39.632464    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:40.632479 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:40.632445    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:40.633021 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:40.632497    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:40.633021 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:40.632574    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:40.633021 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:40.632702    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:41.632656 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:41.632620    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:41.633060 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:41.632743    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:42.632568 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:42.632530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:42.632568 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:42.632552    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:42.633054 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:42.632661    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:42.633054 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:42.632776    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:43.632411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:43.632369    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:43.632627 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:43.632519    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:44.228658 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:44.228614    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:44.229085 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.228748    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:44.229085 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.228809    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:15:00.228791687 +0000 UTC m=+34.129494329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:14:44.328963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:44.328928    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:44.329137 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.329053    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:44.329137 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.329126    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:15:00.329105626 +0000 UTC m=+34.229808268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:14:44.429918 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:44.429889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:44.430074 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.430018    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:14:44.430074 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.430036    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:14:44.430074 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.430050    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:44.430186 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.430105    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:15:00.430088288 +0000 UTC m=+34.330790927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:14:44.631844 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:44.631810    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:44.632005 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.631942    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:44.632061 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:44.631993    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:44.632167 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:44.632136    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:45.632470 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:45.632434    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:45.632935 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:45.632580    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:46.634756 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:46.634726    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:46.635142 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:46.634857    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:46.635142 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:46.634913    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:46.635142 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:46.635022    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:47.631825 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:47.631789    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:47.631997 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:47.631921    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:48.632879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:48.632849    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:48.633476 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:48.632862    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:48.633476 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:48.632993    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:48.633476 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:48.633062    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:49.632781 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.632319    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:49.632933 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:49.632904    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:49.788999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.788921    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal" event={"ID":"121fbd70dde46c6dca67ebc08b171187","Type":"ContainerStarted","Data":"9f0e1ba89b2e9d022f3d67d5485f3b3ca229230f549ef977900ef286d413edad"}
Jul 13 20:14:49.790272 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.790252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-htbsc" event={"ID":"98a95222-55d4-4845-a53a-d00cecd8f210","Type":"ContainerStarted","Data":"6998605676f7ee32e1adfa175426cde2620b1970e56cb8fad10744e7d86ab0c8"}
Jul 13 20:14:49.791704 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.791583    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4nkxs" event={"ID":"33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34","Type":"ContainerStarted","Data":"e60474444706487c11419dbbd49769a652f6035fe8cd885a6f3ba1a40f19733f"}
Jul 13 20:14:49.793419 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.793374    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4h87g" event={"ID":"2efc0e75-f975-4046-b42f-c142d7ed511b","Type":"ContainerStarted","Data":"409c39bf404acb090f2cf1906a3d4e7e7171d2088085cba7557335259d73e2c3"}
Jul 13 20:14:49.794870 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.794847    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nn7d" event={"ID":"9199a222-defd-4294-b6c6-86341b1bc881","Type":"ContainerStarted","Data":"98f4f6f02d141f0f07de5f64d7f23bee7749197cdd2811608359f820894124c7"}
Jul 13 20:14:49.796192 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.796174    2584 generic.go:356] "Generic (PLEG): container finished" podID="45a5d5fc-dc3a-4d65-b776-21e0697f615d" containerID="77bc7a4e7a6a34d3694c22dbe07f121ea8e283442e02cabec59fc1581b30e847" exitCode=0
Jul 13 20:14:49.796281 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.796232    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerDied","Data":"77bc7a4e7a6a34d3694c22dbe07f121ea8e283442e02cabec59fc1581b30e847"}
Jul 13 20:14:49.797849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.797824    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns" event={"ID":"dc4ac74e-a556-44b9-b935-0136456b2898","Type":"ContainerStarted","Data":"c142d0f9b3002f62a2bc775634ab4a77018e2dc92104026e51b6b04d03391db9"}
Jul 13 20:14:49.800419 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.800377    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"6e101a5f66e385dac4f7bf2660025795198549fbe3ca48876895aa346fb06e84"}
Jul 13 20:14:49.800525 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.800452    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"61ebdf768b252bfc59ecf15ca99949bc095354a66f3b430f554cf8e3e04277f9"}
Jul 13 20:14:49.800525 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.800511    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"112a613f4ded859193b338f5206d33d7ccf6f4720d3748379a6e8a84781352ad"}
Jul 13 20:14:49.800613 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.800542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"88953328a840cf70648e900a6f098d830bf4bf4c86841a6ff98fc2f736a08797"}
Jul 13 20:14:49.801626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.801609    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6s44f" event={"ID":"fd8e839d-4afb-4c79-b577-6a8a6b3170ca","Type":"ContainerStarted","Data":"0157dcf012cb849fbf22acfc5a352da47d5c89cab5494df947db120dfd26154e"}
Jul 13 20:14:49.808715 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.808678    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-62.ec2.internal" podStartSLOduration=21.808666895 podStartE2EDuration="21.808666895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:49.808377176 +0000 UTC m=+23.709079838" watchObservedRunningTime="2026-07-13 20:14:49.808666895 +0000 UTC m=+23.709369556"
Jul 13 20:14:49.847675 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.847637    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6s44f" podStartSLOduration=2.643638973 podStartE2EDuration="22.847627129s" totalImagesPullingTime="20.203988156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:49.82825719 +0000 UTC m=+23.728959851" watchObservedRunningTime="2026-07-13 20:14:49.847627129 +0000 UTC m=+23.748329784"
Jul 13 20:14:49.847799 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.847785    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2nn7d" podStartSLOduration=2.634071767 podStartE2EDuration="22.847781971s" totalImagesPullingTime="20.213710204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:49.847732963 +0000 UTC m=+23.748435636" watchObservedRunningTime="2026-07-13 20:14:49.847781971 +0000 UTC m=+23.748484670"
Jul 13 20:14:49.891199 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.891160    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4nkxs" podStartSLOduration=2.685691781 podStartE2EDuration="22.891145666s" totalImagesPullingTime="20.205453885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:49.891137002 +0000 UTC m=+23.791839776" watchObservedRunningTime="2026-07-13 20:14:49.891145666 +0000 UTC m=+23.791848328"
Jul 13 20:14:49.909719 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.909682    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4h87g" podStartSLOduration=2.70056719 podStartE2EDuration="22.90966876s" totalImagesPullingTime="20.20910157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:49.909346469 +0000 UTC m=+23.810049133" watchObservedRunningTime="2026-07-13 20:14:49.90966876 +0000 UTC m=+23.810371422"
Jul 13 20:14:49.930317 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:49.930276    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-htbsc" podStartSLOduration=2.66210157 podStartE2EDuration="22.930260918s" totalImagesPullingTime="20.268159348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:49.929869833 +0000 UTC m=+23.830572495" watchObservedRunningTime="2026-07-13 20:14:49.930260918 +0000 UTC m=+23.830963580"
Jul 13 20:14:50.077455 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.077352    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:50.161146 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.161100    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:50.632619 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.632592    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:50.632788 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.632593    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:50.632788 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:50.632689    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:50.632788 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:50.632775    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:50.813343 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.813304    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"44991e092fed1a2b190ec498733d94f8ff422a27ab94b8a03969414f73ef8d89"}
Jul 13 20:14:50.813759 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.813453    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"45cd922cb74b4f4d1033b6c51880efc04c88672002615cf0401cb382e1814849"}
Jul 13 20:14:50.813870 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.813849    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 20:14:50.815306 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.815280    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4vzzb" event={"ID":"a5d6166d-6f7c-4637-b1eb-42687ded1a6b","Type":"ContainerStarted","Data":"c6cbdf57682974d329bd607914dab4ba55c1ffc5e639e25a827d24e01f0b7a7c"}
Jul 13 20:14:50.831407 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:50.831360    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4vzzb" podStartSLOduration=3.622405798 podStartE2EDuration="23.831345386s" totalImagesPullingTime="20.208939588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:50.831068463 +0000 UTC m=+24.731771126" watchObservedRunningTime="2026-07-13 20:14:50.831345386 +0000 UTC m=+24.732048049"
Jul 13 20:14:51.062551 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.062468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:51.167324 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.167271    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:51.268241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.268205    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:51.561555 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.561452    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T20:14:50.813866632Z","UUID":"931dfdd4-718e-4e9c-8fac-af4735497f68","Handler":null,"Name":"","Endpoint":""}
Jul 13 20:14:51.563436 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.563414    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 20:14:51.563567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.563458    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 20:14:51.632131 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.632100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:51.632304 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:51.632219    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:51.691706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.691675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:51.770055 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.770024    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:51.819952 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.819877    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns" event={"ID":"dc4ac74e-a556-44b9-b935-0136456b2898","Type":"ContainerStarted","Data":"8a673af6482fe6c4a95d7d820ae60b3f330086ac1f656ba93b5033a4c7d4c27a"}
Jul 13 20:14:51.850434 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.850386    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:51.924048 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:51.924024    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:52.632553 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.632358    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:52.632711 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.632358    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:52.632711 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:52.632638    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:52.632807 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:52.632712    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:52.736801 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.736763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:52.824682 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.824642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns" event={"ID":"dc4ac74e-a556-44b9-b935-0136456b2898","Type":"ContainerStarted","Data":"be44eaf730f9b81de58d2e21c15f463f2a7d62594e94768d6f27c3854a550495"}
Jul 13 20:14:52.828019 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.827990    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"1e2b75c385c5ac7747e1c1a606cb750e44160c64bb60c2e1c9c42ec489a22b8e"}
Jul 13 20:14:52.837923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.837863    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:52.853279 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.853223    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gpmns" podStartSLOduration=3.274529783 podStartE2EDuration="25.85320661s" totalImagesPullingTime="22.578676827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:52.852889572 +0000 UTC m=+26.753592235" watchObservedRunningTime="2026-07-13 20:14:52.85320661 +0000 UTC m=+26.753909274"
Jul 13 20:14:52.938223 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:52.938149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:53.632569 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:53.632537    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:53.632752 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:53.632649    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:53.634728 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:53.634703    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:53.635292 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:53.635273    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:53.831143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:53.831108    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:53.831684 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:53.831547    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6s44f"
Jul 13 20:14:53.907847 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:53.907769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:54.632494 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.632298    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:54.632645 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.632298    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:54.632645 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:54.632581    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:54.632713 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:54.632651    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:54.834006 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.833927    2584 generic.go:356] "Generic (PLEG): container finished" podID="45a5d5fc-dc3a-4d65-b776-21e0697f615d" containerID="f4a6cdafaf6fa5c961c3683e68821b1091df8aea1666d6ab22368f9c27dd2473" exitCode=0
Jul 13 20:14:54.834737 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.834006    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerDied","Data":"f4a6cdafaf6fa5c961c3683e68821b1091df8aea1666d6ab22368f9c27dd2473"}
Jul 13 20:14:54.837755 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.837730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" event={"ID":"9e75521d-6024-430e-acc2-4ea6407907e7","Type":"ContainerStarted","Data":"10f5f4137a7d99cf64212b839c078f5e592364dbe217ece5444b4161f5ca93bb"}
Jul 13 20:14:54.837987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.837970    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:54.838060 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.838029    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:54.854123 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.854103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:54.854237 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.854202    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:54.888999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:54.888952    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn" podStartSLOduration=7.588925646 podStartE2EDuration="27.888939338s" totalImagesPullingTime="20.300013692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:14:54.887310309 +0000 UTC m=+28.788012971" watchObservedRunningTime="2026-07-13 20:14:54.888939338 +0000 UTC m=+28.789641999"
Jul 13 20:14:55.632033 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:55.632001    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:55.632217 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:55.632095    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:55.839637 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:55.839609    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 13 20:14:56.229982 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.229773    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gvtll"]
Jul 13 20:14:56.230130 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.230046    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:56.230168 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:56.230149    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:56.232432 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.232388    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4tvtl"]
Jul 13 20:14:56.232580 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.232527    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:56.232649 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:56.232622    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:56.233108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.233087    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vxjq"]
Jul 13 20:14:56.233205 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.233191    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:56.233320 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:56.233298    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:56.479416 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.479369    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:14:56.708497 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.708450    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:56.844048 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.844018    2584 generic.go:356] "Generic (PLEG): container finished" podID="45a5d5fc-dc3a-4d65-b776-21e0697f615d" containerID="c6a715d7b8c6d2931044d48db34d19d29b2dfc98794060574830c38b3c050716" exitCode=0
Jul 13 20:14:56.844486 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:56.844099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerDied","Data":"c6a715d7b8c6d2931044d48db34d19d29b2dfc98794060574830c38b3c050716"}
Jul 13 20:14:57.456572 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.456536    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:57.559806 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.559772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:57.632284 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.632249    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:57.632490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.632249    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:57.632490 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:57.632351    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:57.632622 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:57.632484    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:57.632622 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.632249    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:57.632622 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:57.632581    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:57.637861 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.637839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:57.857266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.857232    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:57.957832 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:57.957804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:58.180850 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:58.180760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:58.249430 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:58.249362    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:58.640293 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:58.640259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:58.741637 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:58.741610    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:58.780968 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:58.780949    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4nkxs_33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34/dns-node-resolver/0.log"
Jul 13 20:14:58.911049 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:58.911022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:59.010265 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.010238    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:14:59.632532 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.632494    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:14:59.632704 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.632604    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:14:59.632704 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:59.632615    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:14:59.632704 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.632627    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:14:59.632704 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:59.632682    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:14:59.632881 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:14:59.632777    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:14:59.780864 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.780839    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4h87g_2efc0e75-f975-4046-b42f-c142d7ed511b/node-ca/0.log"
Jul 13 20:14:59.854250 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.854221    2584 generic.go:356] "Generic (PLEG): container finished" podID="45a5d5fc-dc3a-4d65-b776-21e0697f615d" containerID="9e4872bafddb2bad1810ae1ddb44b29b59514a18b1faf3e8f9cb58ff3dbf20c3" exitCode=0
Jul 13 20:14:59.854408 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:14:59.854263    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerDied","Data":"9e4872bafddb2bad1810ae1ddb44b29b59514a18b1faf3e8f9cb58ff3dbf20c3"}
Jul 13 20:15:00.256210 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:00.256172    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:15:00.256652 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.256330    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:15:00.256652 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.256413    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret podName:b88f467e-b9f0-403b-a2f4-f08a7980c965 nodeName:}" failed. No retries permitted until 2026-07-13 20:15:32.256378175 +0000 UTC m=+66.157080832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret") pod "global-pull-secret-syncer-gvtll" (UID: "b88f467e-b9f0-403b-a2f4-f08a7980c965") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:15:00.357410 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:00.357363    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:15:00.357600 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.357529    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:15:00.357600 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.357593    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs podName:db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0 nodeName:}" failed. No retries permitted until 2026-07-13 20:15:32.357577088 +0000 UTC m=+66.258279727 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs") pod "network-metrics-daemon-2vxjq" (UID: "db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:15:00.458163 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:00.458127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:00.458353 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.458312    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:15:00.458353 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.458336    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:15:00.458353 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.458348    2584 projected.go:196] Error preparing data for projected volume kube-api-access-4tfbb for pod openshift-network-diagnostics/network-check-target-4tvtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:15:00.458528 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:00.458437    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb podName:827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba nodeName:}" failed. No retries permitted until 2026-07-13 20:15:32.458414938 +0000 UTC m=+66.359117601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4tfbb" (UniqueName: "kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb") pod "network-check-target-4tvtl" (UID: "827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:15:00.524174 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:00.524072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:01.601332 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:01.601142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:01.632041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:01.632014    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:15:01.632041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:01.632035    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:15:01.632232 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:01.632132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:01.632232 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:01.632138    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vxjq" podUID="db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0"
Jul 13 20:15:01.632232 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:01.632191    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gvtll" podUID="b88f467e-b9f0-403b-a2f4-f08a7980c965"
Jul 13 20:15:01.632375 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:01.632289    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tvtl" podUID="827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba"
Jul 13 20:15:01.920287 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:01.920217    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-62.ec2.internal" event="NodeReady"
Jul 13 20:15:01.920482 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:01.920376    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 20:15:02.327933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:02.327827    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:02.498472 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:02.498432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:02.649163 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:02.649128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:02.855515 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:02.855480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:02.952038 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:02.951895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:03.050923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.050880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:03.149753 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.149719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:03.250685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.250595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:03.587925 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.587834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:03.631928 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.631901    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:15:03.632095 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.631907    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:15:03.632095 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.631913    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:03.635027 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.634969    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qj7bp\""
Jul 13 20:15:03.635027 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.635004    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 20:15:03.636189 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.636166    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 20:15:03.636301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.636193    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 20:15:03.636301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.636233    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 20:15:03.636301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.636269    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fv6wc\""
Jul 13 20:15:03.846847 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:03.846787    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:04.601245 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:04.601200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:04.660440 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:04.660387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:04.720342 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:04.720284    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:04.828388 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:04.828358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:04.868041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:04.867965    2584 generic.go:356] "Generic (PLEG): container finished" podID="45a5d5fc-dc3a-4d65-b776-21e0697f615d" containerID="1b68e69b51cfb9ed4b0bc239516b107d87502b152b046c6f56e541109f20fe88" exitCode=0
Jul 13 20:15:04.868041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:04.868027    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerDied","Data":"1b68e69b51cfb9ed4b0bc239516b107d87502b152b046c6f56e541109f20fe88"}
Jul 13 20:15:05.024185 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:05.024152    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:05.871127 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:05.871088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:05.875233 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:05.875202    2584 generic.go:356] "Generic (PLEG): container finished" podID="45a5d5fc-dc3a-4d65-b776-21e0697f615d" containerID="cc88d55adec6037a0eb2292d13143e3b37a249869635aeeade67dd14d09dd92c" exitCode=0
Jul 13 20:15:05.875449 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:05.875429    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerDied","Data":"cc88d55adec6037a0eb2292d13143e3b37a249869635aeeade67dd14d09dd92c"}
Jul 13 20:15:06.021967 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.021938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:06.233041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.233000    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:06.424920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.424877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:06.624002 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.623975    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:06.690246 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.690215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:06.841775 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.841742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:06.883075 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.882998    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" event={"ID":"45a5d5fc-dc3a-4d65-b776-21e0697f615d","Type":"ContainerStarted","Data":"3ee8e73b090e4ecab1ccfc0140e3db81dab154c51d9ee022f92665d6c259cbaa"}
Jul 13 20:15:06.912498 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:06.912350    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wdwzm" podStartSLOduration=8.308702224 podStartE2EDuration="39.912331589s" totalImagesPullingTime="31.603629365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:06.91186236 +0000 UTC m=+40.812565022" watchObservedRunningTime="2026-07-13 20:15:06.912331589 +0000 UTC m=+40.813034252"
Jul 13 20:15:07.034496 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:07.034466    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:07.394876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:07.394845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:07.494853 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:07.494810    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:07.859498 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:07.859466    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:08.026454 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:08.026425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:08.169118 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:08.169026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:08.270368 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:08.270325    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:08.627866 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:08.627829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:08.757087 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:08.757050    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:08.875463 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:08.875423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:09.227219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:09.227183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:09.525157 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:09.525065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:09.627586 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:09.627551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:09.822449 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:09.822359    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:10.232528 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:10.232492    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:10.636886 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:10.636857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:10.829365 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:10.829330    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:11.222050 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:11.222013    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:11.629319 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:11.629286    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:12.265255 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:12.265215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:12.619872 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:12.619839    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:12.822576 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:12.822542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:13.025190 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:13.025109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:13.488892 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:13.488857    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:18.100439 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:18.100380    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:18.156236 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:18.156196    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:22.291978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.291945    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d85fc568b-kjb6j"]
Jul 13 20:15:22.325270 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.325233    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d85fc568b-kjb6j"]
Jul 13 20:15:22.325270 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.325277    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"]
Jul 13 20:15:22.325504 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.325353    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.328925 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.328900    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 20:15:22.330521 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.330503    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 20:15:22.332800 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.332780    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 20:15:22.332946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.332929    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mwnds\""
Jul 13 20:15:22.335933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.335913    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 20:15:22.338836 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.338817    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"]
Jul 13 20:15:22.338948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.338930    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.341971 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.341956    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 20:15:22.342281 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.342265    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 20:15:22.342705 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.342687    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 13 20:15:22.342818 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.342797    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 20:15:22.354951 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.354933    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"]
Jul 13 20:15:22.355074 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.355058    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.360993 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.360949    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 20:15:22.361286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.361266    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 20:15:22.361664 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.361554    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 20:15:22.362766 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.362748    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 20:15:22.370938 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.370919    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wd8gn"]
Jul 13 20:15:22.371029 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.371016    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.374077 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.374059    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 13 20:15:22.374197 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.374182    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-74np4\""
Jul 13 20:15:22.391441 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.391420    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"]
Jul 13 20:15:22.391526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.391446    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"]
Jul 13 20:15:22.391526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.391457    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wd8gn"]
Jul 13 20:15:22.391526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.391477    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"]
Jul 13 20:15:22.391526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.391490    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d85fc568b-kjb6j"]
Jul 13 20:15:22.391526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.391521    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.391701 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:15:22.391619    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lc6rs registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lc6rs registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j" podUID="3247f663-63e9-4c96-8d1e-aa5b2497570d"
Jul 13 20:15:22.396771 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.396753    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 20:15:22.396848 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.396822    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jfzbg\""
Jul 13 20:15:22.396992 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.396980    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 20:15:22.409910 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.409884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-image-registry-private-configuration\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.409999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.409911    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3247f663-63e9-4c96-8d1e-aa5b2497570d-ca-trust-extracted\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.409999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.409928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.409999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.409945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn86d\" (UniqueName: \"kubernetes.io/projected/1207a1c8-04fb-4b19-8893-6abc431efddf-kube-api-access-gn86d\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.409999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.409990    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-tmp\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.410154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410021    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1207a1c8-04fb-4b19-8893-6abc431efddf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.410154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410048    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-bound-sa-token\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.410154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410079    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bd3172-a17f-4910-b38c-c79719d75df4-tmp-dir\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.410154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410133    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-installation-pull-secrets\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.410154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410168    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-tls\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410192    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf7gq\" (UniqueName: \"kubernetes.io/projected/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-kube-api-access-zf7gq\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jscrr\" (UniqueName: \"kubernetes.io/projected/830c1c66-62dc-488d-8e17-555de9da3160-kube-api-access-jscrr\") pod \"managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4\" (UID: \"830c1c66-62dc-488d-8e17-555de9da3160\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410229    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-hub\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410261    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-trusted-ca\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-ca\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.410311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410305    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-certificates\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.410535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74bd3172-a17f-4910-b38c-c79719d75df4-config-volume\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.410535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/830c1c66-62dc-488d-8e17-555de9da3160-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4\" (UID: \"830c1c66-62dc-488d-8e17-555de9da3160\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.410535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-klusterlet-config\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.410535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410376    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6rs\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-kube-api-access-lc6rs\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.410535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74bd3172-a17f-4910-b38c-c79719d75df4-metrics-tls\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.410535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.410433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5cr\" (UniqueName: \"kubernetes.io/projected/74bd3172-a17f-4910-b38c-c79719d75df4-kube-api-access-ww5cr\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.438298 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.438279    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qncsw"]
Jul 13 20:15:22.469134 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.469112    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fww5j"]
Jul 13 20:15:22.469227 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.469216    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.472493 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.472468    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 20:15:22.472682 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.472666    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 20:15:22.472758 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.472744    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 20:15:22.472828 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.472783    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2rtsn\""
Jul 13 20:15:22.490594 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.490570    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:22.492871 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.492856    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qncsw"]
Jul 13 20:15:22.492928 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.492889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fww5j"]
Jul 13 20:15:22.492977 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.492966    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.496364 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.496345    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 20:15:22.496550 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.496515    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 20:15:22.496636 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.496548    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 20:15:22.496636 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.496558    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 20:15:22.496873 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.496857    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vgmz9\""
Jul 13 20:15:22.511679 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6rs\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-kube-api-access-lc6rs\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.511760 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511682    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74bd3172-a17f-4910-b38c-c79719d75df4-metrics-tls\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.511760 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66f0e938-3e7f-4e3b-852f-b6998d68711f-cert\") pod \"ingress-canary-qncsw\" (UID: \"66f0e938-3e7f-4e3b-852f-b6998d68711f\") " pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.511760 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511721    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf7gq\" (UniqueName: \"kubernetes.io/projected/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-kube-api-access-zf7gq\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.511760 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511741    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-image-registry-private-configuration\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.511920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511838    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-crio-socket\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.511920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511872    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-tmp\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.511920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wcb\" (UniqueName: \"kubernetes.io/projected/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-kube-api-access-q8wcb\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.511920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511908    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bd3172-a17f-4910-b38c-c79719d75df4-tmp-dir\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.512104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511930    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-hub\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.512104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511970    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74bd3172-a17f-4910-b38c-c79719d75df4-config-volume\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.512104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.511994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-klusterlet-config\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.512104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5cr\" (UniqueName: \"kubernetes.io/projected/74bd3172-a17f-4910-b38c-c79719d75df4-kube-api-access-ww5cr\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.512104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512090    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3247f663-63e9-4c96-8d1e-aa5b2497570d-ca-trust-extracted\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.512337 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512118    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn86d\" (UniqueName: \"kubernetes.io/projected/1207a1c8-04fb-4b19-8893-6abc431efddf-kube-api-access-gn86d\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.512337 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512180    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-data-volume\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.512337 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512230    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jscrr\" (UniqueName: \"kubernetes.io/projected/830c1c66-62dc-488d-8e17-555de9da3160-kube-api-access-jscrr\") pod \"managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4\" (UID: \"830c1c66-62dc-488d-8e17-555de9da3160\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.512337 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512265    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74bd3172-a17f-4910-b38c-c79719d75df4-tmp-dir\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.512337 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.512337 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512310    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-tmp\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512358    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512419    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1207a1c8-04fb-4b19-8893-6abc431efddf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512446    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-bound-sa-token\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-installation-pull-secrets\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512549    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-tls\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512578    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512605    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjl22\" (UniqueName: \"kubernetes.io/projected/66f0e938-3e7f-4e3b-852f-b6998d68711f-kube-api-access-rjl22\") pod \"ingress-canary-qncsw\" (UID: \"66f0e938-3e7f-4e3b-852f-b6998d68711f\") " pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512635    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-trusted-ca\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.512670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512663    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-ca\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.513184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/74bd3172-a17f-4910-b38c-c79719d75df4-config-volume\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.513184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.513031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3247f663-63e9-4c96-8d1e-aa5b2497570d-ca-trust-extracted\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.513884 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.512691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-certificates\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.513884 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.513765    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/830c1c66-62dc-488d-8e17-555de9da3160-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4\" (UID: \"830c1c66-62dc-488d-8e17-555de9da3160\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.514081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.514056    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1207a1c8-04fb-4b19-8893-6abc431efddf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.517344 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.517319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-trusted-ca\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.517762 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.517739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-klusterlet-config\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.517911 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.517739    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-certificates\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.518006 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.517783    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/74bd3172-a17f-4910-b38c-c79719d75df4-metrics-tls\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.518075 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.517832    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.518075 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.517907    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-installation-pull-secrets\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.518180 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.518095    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-hub\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.518233 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.518222    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-image-registry-private-configuration\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.518564 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.518542    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.519591 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.519569    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-tls\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.519730 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.519704    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/830c1c66-62dc-488d-8e17-555de9da3160-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4\" (UID: \"830c1c66-62dc-488d-8e17-555de9da3160\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.520021 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.520004    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6647c786d4-5x7qc"]
Jul 13 20:15:22.520146 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.520129    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1207a1c8-04fb-4b19-8893-6abc431efddf-ca\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.533165 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.533141    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn86d\" (UniqueName: \"kubernetes.io/projected/1207a1c8-04fb-4b19-8893-6abc431efddf-kube-api-access-gn86d\") pod \"cluster-proxy-proxy-agent-59f9c686f6-gvpc7\" (UID: \"1207a1c8-04fb-4b19-8893-6abc431efddf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.533564 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.533362    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf7gq\" (UniqueName: \"kubernetes.io/projected/e4e88715-cdaa-43a2-aec1-5e5a4529cf8d-kube-api-access-zf7gq\") pod \"klusterlet-addon-workmgr-59bdd9878b-v54wg\" (UID: \"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.533656 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.533605    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5cr\" (UniqueName: \"kubernetes.io/projected/74bd3172-a17f-4910-b38c-c79719d75df4-kube-api-access-ww5cr\") pod \"dns-default-wd8gn\" (UID: \"74bd3172-a17f-4910-b38c-c79719d75df4\") " pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.533951 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.533911    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6rs\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-kube-api-access-lc6rs\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.534064 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.534007    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jscrr\" (UniqueName: \"kubernetes.io/projected/830c1c66-62dc-488d-8e17-555de9da3160-kube-api-access-jscrr\") pod \"managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4\" (UID: \"830c1c66-62dc-488d-8e17-555de9da3160\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.537590 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.537570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-bound-sa-token\") pod \"image-registry-7d85fc568b-kjb6j\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") " pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.543779 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.543712    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6647c786d4-5x7qc"]
Jul 13 20:15:22.543888 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.543805    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.588293 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.588264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:22.615007 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.614986    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02e6065a-a51f-4f12-9e99-77482bc178a6-ca-trust-extracted\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615084 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615015    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615084 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615033    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjl22\" (UniqueName: \"kubernetes.io/projected/66f0e938-3e7f-4e3b-852f-b6998d68711f-kube-api-access-rjl22\") pod \"ingress-canary-qncsw\" (UID: \"66f0e938-3e7f-4e3b-852f-b6998d68711f\") " pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.615152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66f0e938-3e7f-4e3b-852f-b6998d68711f-cert\") pod \"ingress-canary-qncsw\" (UID: \"66f0e938-3e7f-4e3b-852f-b6998d68711f\") " pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.615152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615116    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-crio-socket\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615152 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615142    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02e6065a-a51f-4f12-9e99-77482bc178a6-trusted-ca\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615188    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wcb\" (UniqueName: \"kubernetes.io/projected/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-kube-api-access-q8wcb\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02e6065a-a51f-4f12-9e99-77482bc178a6-registry-certificates\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlc9h\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-kube-api-access-rlc9h\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-crio-socket\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615306    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02e6065a-a51f-4f12-9e99-77482bc178a6-image-registry-private-configuration\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-registry-tls\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615415    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02e6065a-a51f-4f12-9e99-77482bc178a6-installation-pull-secrets\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615450    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-data-volume\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-bound-sa-token\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.615567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615525    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.615761 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.615715    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-data-volume\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.617383 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.617361    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66f0e938-3e7f-4e3b-852f-b6998d68711f-cert\") pod \"ingress-canary-qncsw\" (UID: \"66f0e938-3e7f-4e3b-852f-b6998d68711f\") " pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.617790 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.617774    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.630387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.630364    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjl22\" (UniqueName: \"kubernetes.io/projected/66f0e938-3e7f-4e3b-852f-b6998d68711f-kube-api-access-rjl22\") pod \"ingress-canary-qncsw\" (UID: \"66f0e938-3e7f-4e3b-852f-b6998d68711f\") " pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.639297 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.639274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wcb\" (UniqueName: \"kubernetes.io/projected/ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb-kube-api-access-q8wcb\") pod \"insights-runtime-extractor-fww5j\" (UID: \"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb\") " pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.648296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.648276    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:22.674318 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.674285    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"
Jul 13 20:15:22.680708 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.680691    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"
Jul 13 20:15:22.681694 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.681654    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:22.710531 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.710504    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:22.716880 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.716847    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-bound-sa-token\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.716985 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.716895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02e6065a-a51f-4f12-9e99-77482bc178a6-ca-trust-extracted\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.716985 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.716955    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02e6065a-a51f-4f12-9e99-77482bc178a6-trusted-ca\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.716985 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.716982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02e6065a-a51f-4f12-9e99-77482bc178a6-registry-certificates\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.717137 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.717055    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlc9h\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-kube-api-access-rlc9h\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.717137 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.717125    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02e6065a-a51f-4f12-9e99-77482bc178a6-image-registry-private-configuration\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.717257 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.717152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-registry-tls\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.717257 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.717187    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02e6065a-a51f-4f12-9e99-77482bc178a6-installation-pull-secrets\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.717444 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.717424    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02e6065a-a51f-4f12-9e99-77482bc178a6-ca-trust-extracted\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.718453 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.717798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02e6065a-a51f-4f12-9e99-77482bc178a6-registry-certificates\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.718453 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.718385    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02e6065a-a51f-4f12-9e99-77482bc178a6-trusted-ca\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.720356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.720337    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02e6065a-a51f-4f12-9e99-77482bc178a6-image-registry-private-configuration\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.720610 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.720479    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02e6065a-a51f-4f12-9e99-77482bc178a6-installation-pull-secrets\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.720929 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.720905    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-registry-tls\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.730218 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.730196    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-bound-sa-token\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.731988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.731968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlc9h\" (UniqueName: \"kubernetes.io/projected/02e6065a-a51f-4f12-9e99-77482bc178a6-kube-api-access-rlc9h\") pod \"image-registry-6647c786d4-5x7qc\" (UID: \"02e6065a-a51f-4f12-9e99-77482bc178a6\") " pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.779010 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.778780    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qncsw"
Jul 13 20:15:22.793681 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.791320    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:22.802084 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.801817    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fww5j"
Jul 13 20:15:22.853814 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.852476    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:22.900362 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.900231    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"]
Jul 13 20:15:22.924933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.924526    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.924933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.924571    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg" event={"ID":"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d","Type":"ContainerStarted","Data":"f4930608ccc83d71f720596ad10fc82770d3b0f49a88232847d7470718090875"}
Jul 13 20:15:22.929481 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.929173    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4"]
Jul 13 20:15:22.934340 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:22.934281    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod830c1c66_62dc_488d_8e17_555de9da3160.slice/crio-c6ad9bd183ae5ab00648426ce9d7481dc6741a5e7c64c1b5118c211cdf270180 WatchSource:0}: Error finding container c6ad9bd183ae5ab00648426ce9d7481dc6741a5e7c64c1b5118c211cdf270180: Status 404 returned error can't find the container with id c6ad9bd183ae5ab00648426ce9d7481dc6741a5e7c64c1b5118c211cdf270180
Jul 13 20:15:22.950236 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.950088    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:22.979323 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.979294    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wd8gn"]
Jul 13 20:15:22.982743 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:22.982714    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bd3172_a17f_4910_b38c_c79719d75df4.slice/crio-ef9db9e8eed7007ad5af5dc9fbdb61d62ec091e7873d5712a3c9c6e883005b61 WatchSource:0}: Error finding container ef9db9e8eed7007ad5af5dc9fbdb61d62ec091e7873d5712a3c9c6e883005b61: Status 404 returned error can't find the container with id ef9db9e8eed7007ad5af5dc9fbdb61d62ec091e7873d5712a3c9c6e883005b61
Jul 13 20:15:22.993810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.993787    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qncsw"]
Jul 13 20:15:22.995596 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:22.995565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:22.998764 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:22.998723    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f0e938_3e7f_4e3b_852f_b6998d68711f.slice/crio-aac6f9ffc386e556d78581372a6585831c5e4b98b7086fc5593b8bed9662219a WatchSource:0}: Error finding container aac6f9ffc386e556d78581372a6585831c5e4b98b7086fc5593b8bed9662219a: Status 404 returned error can't find the container with id aac6f9ffc386e556d78581372a6585831c5e4b98b7086fc5593b8bed9662219a
Jul 13 20:15:23.012410 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.012374    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fww5j"]
Jul 13 20:15:23.015709 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:23.015687    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd7ce5f_3c9d_4e8a_a96a_c3db6d94fbfb.slice/crio-a5838a77b19294058fb48dd26bc58af839a61ee2fb0865a5d2024d595de84b4b WatchSource:0}: Error finding container a5838a77b19294058fb48dd26bc58af839a61ee2fb0865a5d2024d595de84b4b: Status 404 returned error can't find the container with id a5838a77b19294058fb48dd26bc58af839a61ee2fb0865a5d2024d595de84b4b
Jul 13 20:15:23.023891 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.023871    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3247f663-63e9-4c96-8d1e-aa5b2497570d-ca-trust-extracted\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.023981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.023900    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-bound-sa-token\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.023981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.023920    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-trusted-ca\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.023981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.023946    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc6rs\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-kube-api-access-lc6rs\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.024077 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.023981    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-image-registry-private-configuration\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.024077 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024024    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-tls\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.024077 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024061    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-certificates\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.024220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024095    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-installation-pull-secrets\") pod \"3247f663-63e9-4c96-8d1e-aa5b2497570d\" (UID: \"3247f663-63e9-4c96-8d1e-aa5b2497570d\") "
Jul 13 20:15:23.024220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024118    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3247f663-63e9-4c96-8d1e-aa5b2497570d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:15:23.024357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024336    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3247f663-63e9-4c96-8d1e-aa5b2497570d-ca-trust-extracted\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.024452 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024387    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:15:23.024574 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.024535    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:15:23.026309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.026272    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-kube-api-access-lc6rs" (OuterVolumeSpecName: "kube-api-access-lc6rs") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "kube-api-access-lc6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:15:23.026309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.026290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:15:23.026468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.026357    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:15:23.026728 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.026708    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:15:23.026806 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.026727    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3247f663-63e9-4c96-8d1e-aa5b2497570d" (UID: "3247f663-63e9-4c96-8d1e-aa5b2497570d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:15:23.039661 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.039628    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6647c786d4-5x7qc"]
Jul 13 20:15:23.043235 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:23.043204    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e6065a_a51f_4f12_9e99_77482bc178a6.slice/crio-d8bf14c09662b422e0b3bb84a7cf64428a9bb4c4545286e2568bbfa47df2f557 WatchSource:0}: Error finding container d8bf14c09662b422e0b3bb84a7cf64428a9bb4c4545286e2568bbfa47df2f557: Status 404 returned error can't find the container with id d8bf14c09662b422e0b3bb84a7cf64428a9bb4c4545286e2568bbfa47df2f557
Jul 13 20:15:23.111825 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.111768    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7"]
Jul 13 20:15:23.123247 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:23.123221    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1207a1c8_04fb_4b19_8893_6abc431efddf.slice/crio-a454838b4ad365e0c7ad6ac1a2638ec094f48a0161c0dffc9e82b0ab79e79941 WatchSource:0}: Error finding container a454838b4ad365e0c7ad6ac1a2638ec094f48a0161c0dffc9e82b0ab79e79941: Status 404 returned error can't find the container with id a454838b4ad365e0c7ad6ac1a2638ec094f48a0161c0dffc9e82b0ab79e79941
Jul 13 20:15:23.124667 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124652    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-certificates\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.124747 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124669    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-installation-pull-secrets\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.124747 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124684    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-bound-sa-token\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.124747 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124696    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3247f663-63e9-4c96-8d1e-aa5b2497570d-trusted-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.124747 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124704    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6rs\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-kube-api-access-lc6rs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.124747 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124712    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3247f663-63e9-4c96-8d1e-aa5b2497570d-image-registry-private-configuration\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.124747 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.124721    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3247f663-63e9-4c96-8d1e-aa5b2497570d-registry-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:15:23.222877 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.222852    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:23.934629 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.934536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qncsw" event={"ID":"66f0e938-3e7f-4e3b-852f-b6998d68711f","Type":"ContainerStarted","Data":"aac6f9ffc386e556d78581372a6585831c5e4b98b7086fc5593b8bed9662219a"}
Jul 13 20:15:23.940192 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.940120    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4" event={"ID":"830c1c66-62dc-488d-8e17-555de9da3160","Type":"ContainerStarted","Data":"c6ad9bd183ae5ab00648426ce9d7481dc6741a5e7c64c1b5118c211cdf270180"}
Jul 13 20:15:23.944072 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.944017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7" event={"ID":"1207a1c8-04fb-4b19-8893-6abc431efddf","Type":"ContainerStarted","Data":"a454838b4ad365e0c7ad6ac1a2638ec094f48a0161c0dffc9e82b0ab79e79941"}
Jul 13 20:15:23.950975 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.950906    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fww5j" event={"ID":"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb","Type":"ContainerStarted","Data":"a5838a77b19294058fb48dd26bc58af839a61ee2fb0865a5d2024d595de84b4b"}
Jul 13 20:15:23.958413 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.957573    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc" event={"ID":"02e6065a-a51f-4f12-9e99-77482bc178a6","Type":"ContainerStarted","Data":"aa6371c65a88570c2a950f35a2ceeb75d109e73d12d9d69a39653b3f10f2252c"}
Jul 13 20:15:23.958413 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.957614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc" event={"ID":"02e6065a-a51f-4f12-9e99-77482bc178a6","Type":"ContainerStarted","Data":"d8bf14c09662b422e0b3bb84a7cf64428a9bb4c4545286e2568bbfa47df2f557"}
Jul 13 20:15:23.958413 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.958161    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:23.965411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.960192    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d85fc568b-kjb6j"
Jul 13 20:15:23.965411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.960201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wd8gn" event={"ID":"74bd3172-a17f-4910-b38c-c79719d75df4","Type":"ContainerStarted","Data":"ef9db9e8eed7007ad5af5dc9fbdb61d62ec091e7873d5712a3c9c6e883005b61"}
Jul 13 20:15:23.979834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:23.979776    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc" podStartSLOduration=1.979758861 podStartE2EDuration="1.979758861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:23.979637488 +0000 UTC m=+57.880340151" watchObservedRunningTime="2026-07-13 20:15:23.979758861 +0000 UTC m=+57.880461525"
Jul 13 20:15:24.010212 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:24.010181    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d85fc568b-kjb6j"]
Jul 13 20:15:24.013487 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:24.013454    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d85fc568b-kjb6j"]
Jul 13 20:15:24.643071 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:24.642834    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3247f663-63e9-4c96-8d1e-aa5b2497570d" path="/var/lib/kubelet/pods/3247f663-63e9-4c96-8d1e-aa5b2497570d/volumes"
Jul 13 20:15:27.159280 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.159193    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.232032 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.231943    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.389244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.389214    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.519261 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.519166    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.586598 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.586552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.709674 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.709565    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.810265 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.810176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:27.860072 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:27.860041    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cv2hn"
Jul 13 20:15:28.020946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.020913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.114574 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.114546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.260345 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.260307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.316574 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.316522    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.570423 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.570314    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.670766 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.670726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.819616 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.819572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:28.920694 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:28.920647    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:29.070285 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:29.070248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:29.316508 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:29.316422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:29.774509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:29.774335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:29.839848 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:29.839815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:30.086527 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:30.086432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:30.188985 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:30.188951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:32.303745 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.303713    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:15:32.306707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.306686    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 20:15:32.317411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.317346    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88f467e-b9f0-403b-a2f4-f08a7980c965-original-pull-secret\") pod \"global-pull-secret-syncer-gvtll\" (UID: \"b88f467e-b9f0-403b-a2f4-f08a7980c965\") " pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:15:32.404376 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.404341    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:15:32.407222 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.407189    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 20:15:32.417743 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.417722    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0-metrics-certs\") pod \"network-metrics-daemon-2vxjq\" (UID: \"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0\") " pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:15:32.447610 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.447546    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qj7bp\""
Jul 13 20:15:32.452948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.452929    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gvtll"
Jul 13 20:15:32.455040 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.455021    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vxjq"
Jul 13 20:15:32.505740 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.505707    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:32.508556 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.508537    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 20:15:32.519306 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.519286    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 20:15:32.530053 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.530031    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfbb\" (UniqueName: \"kubernetes.io/projected/827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba-kube-api-access-4tfbb\") pod \"network-check-target-4tvtl\" (UID: \"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba\") " pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:32.760939 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.760910    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fv6wc\""
Jul 13 20:15:32.769216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:32.769191    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:33.282523 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.282112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gvtll"]
Jul 13 20:15:33.289730 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:33.289479    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88f467e_b9f0_403b_a2f4_f08a7980c965.slice/crio-ba8df077f7f399dbc5d556d2053cbe6df591f01eca727c477bc8169f0a306733 WatchSource:0}: Error finding container ba8df077f7f399dbc5d556d2053cbe6df591f01eca727c477bc8169f0a306733: Status 404 returned error can't find the container with id ba8df077f7f399dbc5d556d2053cbe6df591f01eca727c477bc8169f0a306733
Jul 13 20:15:33.301448 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.300615    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vxjq"]
Jul 13 20:15:33.305264 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.303908    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4tvtl"]
Jul 13 20:15:33.308280 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:33.308231    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3c0356_91d7_4b9c_9cbd_5d06cdbedbc0.slice/crio-82ec71281e40b08f1f6dc1a9cb6ed0c188cc77172b50df0241ff5f541c39551a WatchSource:0}: Error finding container 82ec71281e40b08f1f6dc1a9cb6ed0c188cc77172b50df0241ff5f541c39551a: Status 404 returned error can't find the container with id 82ec71281e40b08f1f6dc1a9cb6ed0c188cc77172b50df0241ff5f541c39551a
Jul 13 20:15:33.435185 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.435160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:33.496432 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.496376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:33.705983 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.705948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:34.000113 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.999479    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wd8gn" event={"ID":"74bd3172-a17f-4910-b38c-c79719d75df4","Type":"ContainerStarted","Data":"35f674e52586bc173a89d0459a43ca222ff7451c30470c35e60246703756401d"}
Jul 13 20:15:34.000113 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.999591    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wd8gn" event={"ID":"74bd3172-a17f-4910-b38c-c79719d75df4","Type":"ContainerStarted","Data":"c90f16bb0cfaac29efa226955e37134ece8cb91a4c8543bbba5db2d99f5ea548"}
Jul 13 20:15:34.000113 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:33.999623    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:34.000896 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.000858    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vxjq" event={"ID":"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0","Type":"ContainerStarted","Data":"82ec71281e40b08f1f6dc1a9cb6ed0c188cc77172b50df0241ff5f541c39551a"}
Jul 13 20:15:34.002256 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.002229    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4tvtl" event={"ID":"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba","Type":"ContainerStarted","Data":"7b76cf6c25fb3c1d1c36daaa4fcab847b955af4db77a346abe8efe9a40b24ad7"}
Jul 13 20:15:34.004182 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.004135    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gvtll" event={"ID":"b88f467e-b9f0-403b-a2f4-f08a7980c965","Type":"ContainerStarted","Data":"ba8df077f7f399dbc5d556d2053cbe6df591f01eca727c477bc8169f0a306733"}
Jul 13 20:15:34.005902 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.005859    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qncsw" event={"ID":"66f0e938-3e7f-4e3b-852f-b6998d68711f","Type":"ContainerStarted","Data":"d77d70b5eab3bc1898a58380a114f2d93d36e72a8de573602f91a721e98d7b97"}
Jul 13 20:15:34.007962 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.007935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4" event={"ID":"830c1c66-62dc-488d-8e17-555de9da3160","Type":"ContainerStarted","Data":"750994666ba0b6daa534567f8fb5a83a282ac8cef4aa593a841dda915220f844"}
Jul 13 20:15:34.010680 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.010649    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7" event={"ID":"1207a1c8-04fb-4b19-8893-6abc431efddf","Type":"ContainerStarted","Data":"fa13e41441dd86e986f0c9f6d7a87084a0e1b832691d4a106a7f8c8471ac072e"}
Jul 13 20:15:34.010768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.010705    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7" event={"ID":"1207a1c8-04fb-4b19-8893-6abc431efddf","Type":"ContainerStarted","Data":"c9fa63716278e7005cc4fadec25056c73b5b98b040cce09200ee5efd19f9eca4"}
Jul 13 20:15:34.010768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.010730    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7" event={"ID":"1207a1c8-04fb-4b19-8893-6abc431efddf","Type":"ContainerStarted","Data":"209c01b6d69a278e1ff49a349bf60a224eb092446f22e578c988ef4285b3a06a"}
Jul 13 20:15:34.012775 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.012751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fww5j" event={"ID":"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb","Type":"ContainerStarted","Data":"16728f17e0c07060c991a85395a9595b8180ebcf0fd7dfc6e0a71af2fd8d65e4"}
Jul 13 20:15:34.014369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.014348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg" event={"ID":"e4e88715-cdaa-43a2-aec1-5e5a4529cf8d","Type":"ContainerStarted","Data":"c51d517b4709b96d7ec7e9725acaaeb81fed871dfca3828ae231384e6cba530c"}
Jul 13 20:15:34.014972 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.014925    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:34.016573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.016554    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg"
Jul 13 20:15:34.017924 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.017881    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wd8gn" podStartSLOduration=1.927026572 podStartE2EDuration="12.017862958s" totalImagesPullingTime="10.090836386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:34.016524563 +0000 UTC m=+67.917227223" watchObservedRunningTime="2026-07-13 20:15:34.017862958 +0000 UTC m=+67.918565621"
Jul 13 20:15:34.032496 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.032454    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bdd9878b-v54wg" podStartSLOduration=1.870649277 podStartE2EDuration="12.032441648s" totalImagesPullingTime="10.161792371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:34.032125108 +0000 UTC m=+67.932827791" watchObservedRunningTime="2026-07-13 20:15:34.032441648 +0000 UTC m=+67.933144289"
Jul 13 20:15:34.050980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.050927    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f9c686f6-gvpc7" podStartSLOduration=2.100100724 podStartE2EDuration="12.050910621s" totalImagesPullingTime="9.950809897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:34.050108586 +0000 UTC m=+67.950811249" watchObservedRunningTime="2026-07-13 20:15:34.050910621 +0000 UTC m=+67.951613286"
Jul 13 20:15:34.067014 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.066967    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qncsw" podStartSLOduration=1.991871138 podStartE2EDuration="12.066948539s" totalImagesPullingTime="10.075077401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:34.06644232 +0000 UTC m=+67.967144983" watchObservedRunningTime="2026-07-13 20:15:34.066948539 +0000 UTC m=+67.967651201"
Jul 13 20:15:34.086243 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:34.086107    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dd7cb48b6-h7kn4" podStartSLOduration=1.9442202819999999 podStartE2EDuration="12.086089772s" totalImagesPullingTime="10.14186949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:34.08537461 +0000 UTC m=+67.986077272" watchObservedRunningTime="2026-07-13 20:15:34.086089772 +0000 UTC m=+67.986792436"
Jul 13 20:15:35.022285 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:35.022196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fww5j" event={"ID":"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb","Type":"ContainerStarted","Data":"0c90744d4ef1b577b7bedb17bdf850ac78ec079580b2289a66550bcdf1951136"}
Jul 13 20:15:35.027491 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:35.027448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vxjq" event={"ID":"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0","Type":"ContainerStarted","Data":"f35a931b543d9a8a3525dfc79ec64c4cc338ee7b1837f0de4b700d30fbc77b14"}
Jul 13 20:15:35.027608 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:35.027505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vxjq" event={"ID":"db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0","Type":"ContainerStarted","Data":"20b539f9def7b396d0a41f2308dff6eb5a69425322b43dc493d7e482ef902792"}
Jul 13 20:15:35.043829 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:35.043777    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2vxjq" podStartSLOduration=66.711627312 podStartE2EDuration="1m8.043757819s" totalImagesPullingTime="1.332130507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:35.042561356 +0000 UTC m=+68.943264017" watchObservedRunningTime="2026-07-13 20:15:35.043757819 +0000 UTC m=+68.944460485"
Jul 13 20:15:35.166676 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:35.166014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:35.232437 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:35.232405    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:40.049382 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.049332    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fww5j" event={"ID":"ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb","Type":"ContainerStarted","Data":"ce77681a5f607ae2c1a505e91a38f1cd620fd4d7fad03bffd552e6aaf6cdd723"}
Jul 13 20:15:40.050985 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.050957    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4tvtl" event={"ID":"827dd6df-0ac3-4b5d-8adf-a9fe4b6683ba","Type":"ContainerStarted","Data":"ca9361724a54815d5b66acff69eac8f4fb3fc23624e6fa12c9ea824426db038f"}
Jul 13 20:15:40.051109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.051079    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:15:40.052361 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.052303    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gvtll" event={"ID":"b88f467e-b9f0-403b-a2f4-f08a7980c965","Type":"ContainerStarted","Data":"07b7601e141e8af92c0a9326b32abdd5a45d194c3c0ef2d6714df6c5f73c423d"}
Jul 13 20:15:40.069440 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.069371    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fww5j" podStartSLOduration=1.525768052 podStartE2EDuration="18.069354784s" totalImagesPullingTime="16.543586732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:15:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:40.06807263 +0000 UTC m=+73.968775306" watchObservedRunningTime="2026-07-13 20:15:40.069354784 +0000 UTC m=+73.970057451"
Jul 13 20:15:40.082411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.082362    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gvtll" podStartSLOduration=66.550426479 podStartE2EDuration="1m13.082349182s" totalImagesPullingTime="6.531922703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:40.081752558 +0000 UTC m=+73.982455218" watchObservedRunningTime="2026-07-13 20:15:40.082349182 +0000 UTC m=+73.983051838"
Jul 13 20:15:40.097612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.097571    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4tvtl" podStartSLOduration=66.592872262 podStartE2EDuration="1m13.097559295s" totalImagesPullingTime="6.504687033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:40.096764815 +0000 UTC m=+73.997467476" watchObservedRunningTime="2026-07-13 20:15:40.097559295 +0000 UTC m=+73.998261957"
Jul 13 20:15:40.477551 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.477521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:40.577986 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:40.577953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:41.616834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:41.616795    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:41.921475 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:41.921370    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:41.995000 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:41.994954    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:42.492017 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:42.491981    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:42.588195 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:42.588165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:42.857622 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:42.857592    2584 patch_prober.go:28] interesting pod/image-registry-6647c786d4-5x7qc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 13 20:15:42.857959 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:42.857640    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc" podUID="02e6065a-a51f-4f12-9e99-77482bc178a6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:15:43.583280 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.583244    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:43.677918 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.677877    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:43.783707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.783677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:43.832496 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.832462    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6nr4c"]
Jul 13 20:15:43.834783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.834720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.837542 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.837516    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 20:15:43.837667 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.837643    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nkhzr\""
Jul 13 20:15:43.838872 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.838850    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 20:15:43.838963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.838875    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 20:15:43.838963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.838893    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 20:15:43.838963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.838902    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 20:15:43.839210 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.839182    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 20:15:43.870455 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.870428    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:43.897025 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897005    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897124 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897039    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-tls\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897124 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897068    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-sys\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897124 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-textfile\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897141    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-wtmp\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfslt\" (UniqueName: \"kubernetes.io/projected/eac57dfb-d773-40de-bf82-430dcc3601f4-kube-api-access-mfslt\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897203    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eac57dfb-d773-40de-bf82-430dcc3601f4-metrics-client-ca\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897219    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-root\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.897301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.897238    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.953540 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.953516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:43.998207 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998179    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eac57dfb-d773-40de-bf82-430dcc3601f4-metrics-client-ca\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998212    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-root\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998238    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998289    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998297    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-root\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998318    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-tls\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-sys\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998379    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-textfile\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998430    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-wtmp\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998471    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfslt\" (UniqueName: \"kubernetes.io/projected/eac57dfb-d773-40de-bf82-430dcc3601f4-kube-api-access-mfslt\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998485    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-sys\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998631    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-wtmp\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998736    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-textfile\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.998810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998808    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eac57dfb-d773-40de-bf82-430dcc3601f4-metrics-client-ca\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:43.999027 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:43.998976    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:44.000856 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:44.000834    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-tls\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:44.000856 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:44.000852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eac57dfb-d773-40de-bf82-430dcc3601f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:44.009568 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:44.009550    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfslt\" (UniqueName: \"kubernetes.io/projected/eac57dfb-d773-40de-bf82-430dcc3601f4-kube-api-access-mfslt\") pod \"node-exporter-6nr4c\" (UID: \"eac57dfb-d773-40de-bf82-430dcc3601f4\") " pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:44.029783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:44.029763    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wd8gn"
Jul 13 20:15:44.053367 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:44.053337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:44.145270 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:44.145243    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6nr4c"
Jul 13 20:15:44.155408 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:15:44.155373    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac57dfb_d773_40de_bf82_430dcc3601f4.slice/crio-f3695ac7c68782d7f7b6102a72c60eada7e97d3f25fc4bc214e8557527b0bd03 WatchSource:0}: Error finding container f3695ac7c68782d7f7b6102a72c60eada7e97d3f25fc4bc214e8557527b0bd03: Status 404 returned error can't find the container with id f3695ac7c68782d7f7b6102a72c60eada7e97d3f25fc4bc214e8557527b0bd03
Jul 13 20:15:45.070836 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:45.070799    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6nr4c" event={"ID":"eac57dfb-d773-40de-bf82-430dcc3601f4","Type":"ContainerStarted","Data":"f3695ac7c68782d7f7b6102a72c60eada7e97d3f25fc4bc214e8557527b0bd03"}
Jul 13 20:15:45.138302 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:45.138274    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:45.239073 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:45.239035    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:45.979274 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:45.979247    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6647c786d4-5x7qc"
Jul 13 20:15:46.105170 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.105134    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.165190 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.165156    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.357376 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.357339    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.493154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.493123    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.593458 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.593339    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.722675 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.722609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.823684 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.823649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:46.940039 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:46.940010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:47.081623 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:47.081542    2584 generic.go:356] "Generic (PLEG): container finished" podID="eac57dfb-d773-40de-bf82-430dcc3601f4" containerID="ffde2fadd24372d9d75833af96450ea3c8525946f6fc4c8129fef0be7c37e67e" exitCode=0
Jul 13 20:15:47.081740 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:47.081639    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6nr4c" event={"ID":"eac57dfb-d773-40de-bf82-430dcc3601f4","Type":"ContainerDied","Data":"ffde2fadd24372d9d75833af96450ea3c8525946f6fc4c8129fef0be7c37e67e"}
Jul 13 20:15:47.431884 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:47.431851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:48.087442 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.087387    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6nr4c" event={"ID":"eac57dfb-d773-40de-bf82-430dcc3601f4","Type":"ContainerStarted","Data":"51df8ea2986e8024ce842deb38c4dd7e89c088f686af6b30d4f7fb73c5e9c7ad"}
Jul 13 20:15:48.087613 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.087459    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6nr4c" event={"ID":"eac57dfb-d773-40de-bf82-430dcc3601f4","Type":"ContainerStarted","Data":"3b58d9dd3527c35fe9e332871c04447f43f18f9896a68ee549bbe43d869eb9fc"}
Jul 13 20:15:48.117600 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.117575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:48.127681 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.127641    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6nr4c" podStartSLOduration=2.827248415 podStartE2EDuration="5.127627644s" totalImagesPullingTime="2.230521681s" totalInitContainerRuntime="69.857548ms" isStatefulPod=true podCreationTimestamp="2026-07-13 20:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:15:48.126352264 +0000 UTC m=+82.027054925" watchObservedRunningTime="2026-07-13 20:15:48.127627644 +0000 UTC m=+82.028330305"
Jul 13 20:15:48.209638 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.209611    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:48.574663 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.574586    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:48.674730 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:48.674677    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:49.274911 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:49.274869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:50.076145 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:50.076102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:50.166831 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:50.166801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:50.265207 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:50.265183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:52.285604 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:52.285564    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:52.385774 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:52.385736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:54.882311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:54.882276    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:54.983055 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:54.983020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:55.372083 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:55.372043    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:56.421435 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:56.421369    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:59.294269 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:59.294231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:15:59.394469 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:15:59.394439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:00.141004 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:00.140966    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:00.219322 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:00.219290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:00.321960 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:00.321925    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:03.098593 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:03.098560    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:03.156490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:03.156454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:10.261045 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:10.260999    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:10.361437 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:10.361381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:10.419193 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:10.419149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:11.059158 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:11.059130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4tvtl"
Jul 13 20:16:18.109293 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:18.109255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:18.207171 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:18.207142    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:16:36.974899 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:16:36.974867    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:03.091166 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:03.091127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:03.287647 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:03.287613    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:12.026075 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:12.026033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:12.082660 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:12.082623    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:22.550975 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:22.550927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:22.651518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:22.651488    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:22.709872 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:22.709815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:28.451153 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:28.451110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:43.497825 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:43.497789    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:17:43.551578 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:17:43.551548    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:09.388506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:09.388468    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:18.212738 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:18.212704    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:18.338509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:18.338477    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:18.408357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:18.408328    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:27.480808 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:27.480769    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:27.581164 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:27.581124    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:27.635024 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:27.634997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:28.409382 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:28.409341    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:28.477033 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:28.476997    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:30.284150 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:30.284109    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:30.348537 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:30.348506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:33.107265 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:33.107212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:33.208860 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:33.208822    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:33.265171 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:33.265139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:33.490371 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:33.490283    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:33.547297 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:33.547264    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:38.849055 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:38.849022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:38.957820 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:38.957784    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:18:48.818546 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:18:48.818456    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:05.220576 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:05.220524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:05.334194 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:05.334160    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:26.495539 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:26.495513    2584 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 20:19:37.521909 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:37.521873    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:39.397719 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:39.397683    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:41.356443 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:41.356376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:41.455984 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:41.455957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:42.776635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:42.776604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:42.953433 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:42.953374    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:45.214670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:45.214636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:45.647439 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:45.647411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.122552 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:48.122517    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.224150 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:48.224121    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:52.475506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:52.475463    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:53.589050 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:53.589017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:53.642654 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:53.642627    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:57.739629 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:57.739600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:57.838926 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:19:57.838899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:04.315917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:04.315883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:04.382051 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:04.382020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:05.719555 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:05.719523    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:22.346126 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:22.346094    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:22.446817 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:22.446784    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:22.985595 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:22.985558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.219706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.219663    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.329228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.329127    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.390087 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.390060    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"]
Jul 13 20:20:23.392107 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.392093    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.394918 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.394892    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 13 20:20:23.395036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.394944    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 13 20:20:23.395036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.395001    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 13 20:20:23.395036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.395009    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 13 20:20:23.395036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.395023    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n4gfr\""
Jul 13 20:20:23.407280 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.407252    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"]
Jul 13 20:20:23.553064 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.553023    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwx9\" (UniqueName: \"kubernetes.io/projected/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-kube-api-access-lgwx9\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.553188 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.553070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-tmp\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.553188 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.553153    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.553259 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.553196    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-manifests\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.553259 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.553223    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.590809 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.590776    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.654439 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.654386    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwx9\" (UniqueName: \"kubernetes.io/projected/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-kube-api-access-lgwx9\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.654439 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.654443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-tmp\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.654573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.654474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.654573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.654490    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-manifests\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.654573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.654506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.654913 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.654893    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-manifests\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.657028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.657001    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-tmp\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.657114 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.657094    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.657216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.657201    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.663344 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.663325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwx9\" (UniqueName: \"kubernetes.io/projected/e15571ba-9b87-47f3-a1d5-c08cefe8a9c6-kube-api-access-lgwx9\") pod \"opendatahub-operator-controller-manager-7b56dd55f-fb9gq\" (UID: \"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6\") " pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.699055 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.699029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.702032 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.702017    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:23.791047 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.791011    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.841136 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.840954    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"]
Jul 13 20:20:23.843927 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:20:23.843897    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15571ba_9b87_47f3_a1d5_c08cefe8a9c6.slice/crio-18d88a41333d9bd263473bb3035159df459577abd0f0c7bb03ba228b6dd56902 WatchSource:0}: Error finding container 18d88a41333d9bd263473bb3035159df459577abd0f0c7bb03ba228b6dd56902: Status 404 returned error can't find the container with id 18d88a41333d9bd263473bb3035159df459577abd0f0c7bb03ba228b6dd56902
Jul 13 20:20:23.845563 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.845547    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:20:23.890555 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.890529    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.970501 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.970476    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq" event={"ID":"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6","Type":"ContainerStarted","Data":"18d88a41333d9bd263473bb3035159df459577abd0f0c7bb03ba228b6dd56902"}
Jul 13 20:20:23.995678 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:23.995653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:24.113372 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:24.113299    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:24.196122 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:24.196098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:27.987463 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:27.987430    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq" event={"ID":"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6","Type":"ContainerStarted","Data":"0c9cfd1e4793ad815e010f128e3cbac7443ee9834334a733ca9aa2e2802ffb4d"}
Jul 13 20:20:28.992416 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:28.992368    2584 generic.go:356] "Generic (PLEG): container finished" podID="e15571ba-9b87-47f3-a1d5-c08cefe8a9c6" containerID="0c9cfd1e4793ad815e010f128e3cbac7443ee9834334a733ca9aa2e2802ffb4d" exitCode=0
Jul 13 20:20:28.992803 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:28.992455    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq" event={"ID":"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6","Type":"ContainerDied","Data":"0c9cfd1e4793ad815e010f128e3cbac7443ee9834334a733ca9aa2e2802ffb4d"}
Jul 13 20:20:29.998004 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:29.997967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq" event={"ID":"e15571ba-9b87-47f3-a1d5-c08cefe8a9c6","Type":"ContainerStarted","Data":"259c4c881a7e76d25ac3f4e3c66c83ec22408783590d582f8e25528e904afa04"}
Jul 13 20:20:29.998453 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:29.998099    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:30.022974 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:30.022928    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq" podStartSLOduration=2.346468224 podStartE2EDuration="7.022909146s" totalImagesPullingTime="3.62166127s" totalInitContainerRuntime="1.054779652s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:30.021611042 +0000 UTC m=+363.922313715" watchObservedRunningTime="2026-07-13 20:20:30.022909146 +0000 UTC m=+363.923611808"
Jul 13 20:20:31.742778 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:31.742747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:31.843874 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:31.843845    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:35.340861 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:35.340827    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:35.444988 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:35.444960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:40.051603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:40.051572    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:40.151834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:40.151801    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:40.532274 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:40.532240    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:40.624128 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:40.624099    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:41.004061 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:41.004027    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b56dd55f-fb9gq"
Jul 13 20:20:42.773076 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:42.773037    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:42.920325 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:42.920289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:43.193563 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.193533    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:43.534082 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.534009    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-779f44c978-67b7s"]
Jul 13 20:20:43.543480 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.543460    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.546583 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.546565    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 20:20:43.546703 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.546588    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 13 20:20:43.546703 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.546636    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 20:20:43.547923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.547906    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 13 20:20:43.548100 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.548088    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xlwdg\""
Jul 13 20:20:43.549844 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.549822    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779f44c978-67b7s"]
Jul 13 20:20:43.580865 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.580836    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:43.692369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.692346    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259b5d7-2dd2-47f1-a506-e38a528950c8-tls-certs\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.692482 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.692382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2259b5d7-2dd2-47f1-a506-e38a528950c8-tmp\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.692482 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.692463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgzlj\" (UniqueName: \"kubernetes.io/projected/2259b5d7-2dd2-47f1-a506-e38a528950c8-kube-api-access-sgzlj\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.792706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.792648    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259b5d7-2dd2-47f1-a506-e38a528950c8-tls-certs\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.792706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.792693    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2259b5d7-2dd2-47f1-a506-e38a528950c8-tmp\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.793102 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.792873    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgzlj\" (UniqueName: \"kubernetes.io/projected/2259b5d7-2dd2-47f1-a506-e38a528950c8-kube-api-access-sgzlj\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.799423 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.799372    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2259b5d7-2dd2-47f1-a506-e38a528950c8-tmp\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.799992 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.799972    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2259b5d7-2dd2-47f1-a506-e38a528950c8-tls-certs\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.808451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.808430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgzlj\" (UniqueName: \"kubernetes.io/projected/2259b5d7-2dd2-47f1-a506-e38a528950c8-kube-api-access-sgzlj\") pod \"kube-auth-proxy-779f44c978-67b7s\" (UID: \"2259b5d7-2dd2-47f1-a506-e38a528950c8\") " pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.854910 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.854889    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s"
Jul 13 20:20:43.885696 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.885674    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r8xqf"]
Jul 13 20:20:43.887750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.887734    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:43.890622 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.890605    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 13 20:20:43.890730 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.890637    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-nnjh5\""
Jul 13 20:20:43.902623 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.902601    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r8xqf"]
Jul 13 20:20:43.989046 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.989013    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779f44c978-67b7s"]
Jul 13 20:20:43.992461 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:20:43.992431    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2259b5d7_2dd2_47f1_a506_e38a528950c8.slice/crio-7cafb3a8cb3e1025837ede37100976ea673dc74bc5789fbc8f3c41bbc5a3f223 WatchSource:0}: Error finding container 7cafb3a8cb3e1025837ede37100976ea673dc74bc5789fbc8f3c41bbc5a3f223: Status 404 returned error can't find the container with id 7cafb3a8cb3e1025837ede37100976ea673dc74bc5789fbc8f3c41bbc5a3f223
Jul 13 20:20:43.992640 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.992579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:43.994468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.994445    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57m6m\" (UniqueName: \"kubernetes.io/projected/deca5913-494d-4207-8834-c9d128709c3b-kube-api-access-57m6m\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:43.994566 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:43.994483    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deca5913-494d-4207-8834-c9d128709c3b-cert\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.052198 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.052142    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s" event={"ID":"2259b5d7-2dd2-47f1-a506-e38a528950c8","Type":"ContainerStarted","Data":"7cafb3a8cb3e1025837ede37100976ea673dc74bc5789fbc8f3c41bbc5a3f223"}
Jul 13 20:20:44.086127 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.086101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:44.095514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.095489    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57m6m\" (UniqueName: \"kubernetes.io/projected/deca5913-494d-4207-8834-c9d128709c3b-kube-api-access-57m6m\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.095614 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.095521    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deca5913-494d-4207-8834-c9d128709c3b-cert\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.095670 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:20:44.095650    2584 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 13 20:20:44.095731 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:20:44.095719    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deca5913-494d-4207-8834-c9d128709c3b-cert podName:deca5913-494d-4207-8834-c9d128709c3b nodeName:}" failed. No retries permitted until 2026-07-13 20:20:44.595700083 +0000 UTC m=+378.496402733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/deca5913-494d-4207-8834-c9d128709c3b-cert") pod "odh-model-controller-7998c67dc4-r8xqf" (UID: "deca5913-494d-4207-8834-c9d128709c3b") : secret "odh-model-controller-webhook-cert" not found
Jul 13 20:20:44.113768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.113747    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57m6m\" (UniqueName: \"kubernetes.io/projected/deca5913-494d-4207-8834-c9d128709c3b-kube-api-access-57m6m\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.192925 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.192900    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:44.397650 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.397625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:44.598552 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.598529    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deca5913-494d-4207-8834-c9d128709c3b-cert\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.601045 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.601019    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deca5913-494d-4207-8834-c9d128709c3b-cert\") pod \"odh-model-controller-7998c67dc4-r8xqf\" (UID: \"deca5913-494d-4207-8834-c9d128709c3b\") " pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.800169 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.800101    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:44.909143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.909114    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:44.966623 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:44.966574    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r8xqf"]
Jul 13 20:20:44.971529 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:20:44.971504    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeca5913_494d_4207_8834_c9d128709c3b.slice/crio-8dc78eb85a27b4406b1b808c9e357c902498ca4fc92bf10d3891fe673b10e260 WatchSource:0}: Error finding container 8dc78eb85a27b4406b1b808c9e357c902498ca4fc92bf10d3891fe673b10e260: Status 404 returned error can't find the container with id 8dc78eb85a27b4406b1b808c9e357c902498ca4fc92bf10d3891fe673b10e260
Jul 13 20:20:45.047101 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.047072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:45.057881 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.057819    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf" event={"ID":"deca5913-494d-4207-8834-c9d128709c3b","Type":"ContainerStarted","Data":"8dc78eb85a27b4406b1b808c9e357c902498ca4fc92bf10d3891fe673b10e260"}
Jul 13 20:20:45.063011 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.062980    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"]
Jul 13 20:20:45.065129 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.065108    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.068805 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.068751    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 13 20:20:45.068931 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.068841    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h7ncq\""
Jul 13 20:20:45.068931 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.068873    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 13 20:20:45.069161 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.069147    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 13 20:20:45.069556 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.069533    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 13 20:20:45.086155 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.086133    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"]
Jul 13 20:20:45.203932 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.203895    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-token\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.204112 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.203989    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.204112 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.204054    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.204206 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.204123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmz2\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-kube-api-access-mzmz2\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.204206 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.204154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aa6726-845d-4e69-bbb5-01017c7e8a57-local-certs\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.204206 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.204182    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-cacerts\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.204326 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.204207    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.255492 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.255455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:45.305372 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305341    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.305499 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305383    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmz2\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-kube-api-access-mzmz2\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.305499 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305427    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aa6726-845d-4e69-bbb5-01017c7e8a57-local-certs\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.305499 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305448    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-cacerts\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.305499 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305465    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.305691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305506    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-token\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.305691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305664    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.306015 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.305989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.307841 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.307819    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aa6726-845d-4e69-bbb5-01017c7e8a57-local-certs\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.308124 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.308079    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-cacerts\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.308368 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.308351    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.308490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.308463    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.315573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.315551    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-token\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.315779 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.315763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmz2\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-kube-api-access-mzmz2\") pod \"istiod-openshift-gateway-7574459656-lkwfj\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.355402 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.355370    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:45.375823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.375805    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:45.522463 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:45.522437    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"]
Jul 13 20:20:45.524679 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:20:45.524658    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87aa6726_845d_4e69_bbb5_01017c7e8a57.slice/crio-5138f67555a54d07843f2d777856688e34ec520a805997eeca85d093c2834ba9 WatchSource:0}: Error finding container 5138f67555a54d07843f2d777856688e34ec520a805997eeca85d093c2834ba9: Status 404 returned error can't find the container with id 5138f67555a54d07843f2d777856688e34ec520a805997eeca85d093c2834ba9
Jul 13 20:20:46.064702 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:46.064661    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" event={"ID":"87aa6726-845d-4e69-bbb5-01017c7e8a57","Type":"ContainerStarted","Data":"5138f67555a54d07843f2d777856688e34ec520a805997eeca85d093c2834ba9"}
Jul 13 20:20:46.218907 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:46.218870    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:49.650067 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:49.650032    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 13 20:20:49.650487 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:49.650104    2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 13 20:20:50.082531 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.082439    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf" event={"ID":"deca5913-494d-4207-8834-c9d128709c3b","Type":"ContainerStarted","Data":"e0be3cd72547d321e3ef4ebb1cc19b777644212853a5e748718487fda865e9a1"}
Jul 13 20:20:50.082701 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.082540    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:20:50.084201 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.084176    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s" event={"ID":"2259b5d7-2dd2-47f1-a506-e38a528950c8","Type":"ContainerStarted","Data":"4322ec54b3c96d0146e7c52831df0148c748f9c4743a91e9df0613fce2ef1b78"}
Jul 13 20:20:50.086017 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.085995    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" event={"ID":"87aa6726-845d-4e69-bbb5-01017c7e8a57","Type":"ContainerStarted","Data":"e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd"}
Jul 13 20:20:50.086195 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.086181    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:50.087795 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.087772    2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lkwfj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 20:20:50.087895 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.087825    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:20:50.124202 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.124164    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf" podStartSLOduration=2.497857764 podStartE2EDuration="7.124150921s" totalImagesPullingTime="4.626293157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:50.123272912 +0000 UTC m=+384.023975574" watchObservedRunningTime="2026-07-13 20:20:50.124150921 +0000 UTC m=+384.024853561"
Jul 13 20:20:50.197923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.197873    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-779f44c978-67b7s" podStartSLOduration=1.605924082 podStartE2EDuration="7.197858473s" totalImagesPullingTime="5.591934391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:50.163877012 +0000 UTC m=+384.064579689" watchObservedRunningTime="2026-07-13 20:20:50.197858473 +0000 UTC m=+384.098561136"
Jul 13 20:20:50.198074 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.198022    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" podStartSLOduration=1.074744047 podStartE2EDuration="5.198012395s" totalImagesPullingTime="4.123268348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:50.196240938 +0000 UTC m=+384.096943600" watchObservedRunningTime="2026-07-13 20:20:50.198012395 +0000 UTC m=+384.098715057"
Jul 13 20:20:50.497374 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.497329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:50.666236 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:50.666187    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:51.090840 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:51.090811    2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lkwfj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 13 20:20:51.091002 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:51.090861    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:20:51.581996 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:51.581903    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:51.851860 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:51.851826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:51.993860 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:51.993830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:52.643828 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.643795    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"]
Jul 13 20:20:52.646876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.646860    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.649942 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.649919    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 13 20:20:52.650053 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.649975    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:20:52.650053 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.650005    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mqc4w\""
Jul 13 20:20:52.651057 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.651042    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 13 20:20:52.651290 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.651278    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 13 20:20:52.651415 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.651386    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 13 20:20:52.662523 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.662499    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-cert\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.662632 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.662541    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-metrics-cert\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.662632 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.662606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxw6g\" (UniqueName: \"kubernetes.io/projected/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-kube-api-access-jxw6g\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.662748 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.662708    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-manager-config\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.663956 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.663936    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"]
Jul 13 20:20:52.763243 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.763210    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-cert\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.763375 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.763251    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-metrics-cert\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.763375 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.763271    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxw6g\" (UniqueName: \"kubernetes.io/projected/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-kube-api-access-jxw6g\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.763511 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.763458    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-manager-config\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.764048 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.764026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-manager-config\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.765707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.765681    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-cert\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.765790 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.765781    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-metrics-cert\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.786629 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.786610    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxw6g\" (UniqueName: \"kubernetes.io/projected/ac0bacd5-a23d-4e9a-9606-96eb0fe92948-kube-api-access-jxw6g\") pod \"lws-controller-manager-7f676ff564-m2dgv\" (UID: \"ac0bacd5-a23d-4e9a-9606-96eb0fe92948\") " pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:52.956029 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:52.955955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:53.107385 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:53.107354    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"]
Jul 13 20:20:53.110510 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:20:53.110483    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0bacd5_a23d_4e9a_9606_96eb0fe92948.slice/crio-c9f301bef32e58d7f960d07853557496b9e0aef6dcaeb8ed821d56f48ffcedbd WatchSource:0}: Error finding container c9f301bef32e58d7f960d07853557496b9e0aef6dcaeb8ed821d56f48ffcedbd: Status 404 returned error can't find the container with id c9f301bef32e58d7f960d07853557496b9e0aef6dcaeb8ed821d56f48ffcedbd
Jul 13 20:20:54.091301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:54.091259    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:54.104810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:54.104751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv" event={"ID":"ac0bacd5-a23d-4e9a-9606-96eb0fe92948","Type":"ContainerStarted","Data":"c9f301bef32e58d7f960d07853557496b9e0aef6dcaeb8ed821d56f48ffcedbd"}
Jul 13 20:20:55.552626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:55.552598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:55.701675 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:55.701629    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:56.002892 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:56.002856    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:56.112551 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:56.112522    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv" event={"ID":"ac0bacd5-a23d-4e9a-9606-96eb0fe92948","Type":"ContainerStarted","Data":"81378da6bbad53da61e1d6313da1da6bc047b3e560528e1ca133efc537cbe3cf"}
Jul 13 20:20:56.112676 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:56.112589    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:20:56.149480 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:56.149435    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv" podStartSLOduration=1.712595729 podStartE2EDuration="4.149421112s" totalImagesPullingTime="2.436825383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:56.148892765 +0000 UTC m=+390.049595428" watchObservedRunningTime="2026-07-13 20:20:56.149421112 +0000 UTC m=+390.050123768"
Jul 13 20:20:56.186785 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:56.186760    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:56.350704 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:56.350678    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:57.653858 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:57.653829    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"]
Jul 13 20:20:57.654317 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:57.653992    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerName="discovery" containerID="cri-o://e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd" gracePeriod=30
Jul 13 20:20:57.899035 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:57.899004    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:58.004109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004042    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-dns-cert\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.004109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004093    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aa6726-845d-4e69-bbb5-01017c7e8a57-local-certs\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.004321 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004141    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-kubeconfig\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.004321 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004166    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-token\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.004321 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004200    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzmz2\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-kube-api-access-mzmz2\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.004521 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004329    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-cacerts\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.004521 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.004374    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-ca-configmap\") pod \"87aa6726-845d-4e69-bbb5-01017c7e8a57\" (UID: \"87aa6726-845d-4e69-bbb5-01017c7e8a57\") "
Jul 13 20:20:58.005085 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.005054    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:20:58.006554 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.006523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:20:58.006677 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.006523    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aa6726-845d-4e69-bbb5-01017c7e8a57-local-certs" (OuterVolumeSpecName: "local-certs") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:20:58.006839 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.006815    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-token" (OuterVolumeSpecName: "istio-token") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:20:58.006938 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.006879    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:20:58.006982 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.006950    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-cacerts" (OuterVolumeSpecName: "cacerts") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:20:58.007108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.006977    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-kube-api-access-mzmz2" (OuterVolumeSpecName: "kube-api-access-mzmz2") pod "87aa6726-845d-4e69-bbb5-01017c7e8a57" (UID: "87aa6726-845d-4e69-bbb5-01017c7e8a57"). InnerVolumeSpecName "kube-api-access-mzmz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:20:58.105277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105256    2584 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-kubeconfig\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.105277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105275    2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-token\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.105431 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105283    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzmz2\" (UniqueName: \"kubernetes.io/projected/87aa6726-845d-4e69-bbb5-01017c7e8a57-kube-api-access-mzmz2\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.105431 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105292    2584 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-cacerts\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.105431 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105300    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-ca-configmap\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.105431 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105309    2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87aa6726-845d-4e69-bbb5-01017c7e8a57-istio-csr-dns-cert\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.105431 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.105317    2584 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aa6726-845d-4e69-bbb5-01017c7e8a57-local-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:20:58.121026 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.120996    2584 generic.go:356] "Generic (PLEG): container finished" podID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerID="e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd" exitCode=0
Jul 13 20:20:58.121113 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.121028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" event={"ID":"87aa6726-845d-4e69-bbb5-01017c7e8a57","Type":"ContainerDied","Data":"e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd"}
Jul 13 20:20:58.121113 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.121045    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"
Jul 13 20:20:58.121182 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.121112    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj" event={"ID":"87aa6726-845d-4e69-bbb5-01017c7e8a57","Type":"ContainerDied","Data":"5138f67555a54d07843f2d777856688e34ec520a805997eeca85d093c2834ba9"}
Jul 13 20:20:58.121182 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.121143    2584 scope.go:117] "RemoveContainer" containerID="e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd"
Jul 13 20:20:58.129800 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.129775    2584 scope.go:117] "RemoveContainer" containerID="e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd"
Jul 13 20:20:58.130071 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:20:58.130048    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd\": container with ID starting with e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd not found: ID does not exist" containerID="e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd"
Jul 13 20:20:58.130120 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.130079    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd"} err="failed to get container status \"e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd\": rpc error: code = NotFound desc = could not find container \"e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd\": container with ID starting with e66b89bf83b05a92f6422608c9e95e665193de7159dd8b7d04db5aa34e007bdd not found: ID does not exist"
Jul 13 20:20:58.142342 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.142315    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"]
Jul 13 20:20:58.146728 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.146707    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lkwfj"]
Jul 13 20:20:58.637147 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:58.637099    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" path="/var/lib/kubelet/pods/87aa6726-845d-4e69-bbb5-01017c7e8a57/volumes"
Jul 13 20:20:59.091358 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:20:59.091327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:01.091989 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:01.091964    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r8xqf"
Jul 13 20:21:07.117779 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:07.117747    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f676ff564-m2dgv"
Jul 13 20:21:43.737234 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:43.737198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:43.838123 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:43.838079    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:44.298909 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:44.298875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:44.396917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:44.396883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:44.503691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:44.503659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:45.285409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:45.285358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:45.451567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:45.451534    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:45.587456 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:45.587423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:45.647964 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:21:45.647927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:12.697880 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:12.697851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:13.293564 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:13.293527    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:14.335680 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:14.335638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:14.450160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:14.450128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:17.983228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:17.983190    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:18.353647 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:18.353621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:18.453175 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:18.453125    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:21.003562 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:21.003523    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:22.336274 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:22.336239    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:23.791158 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:23.791122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:24.726690 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:24.726653    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:26.173168 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:26.173134    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:28.142290 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:28.142237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:28.276239 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:28.276199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:28.765892 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:28.765860    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:28.858701 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:28.858670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:28.999188 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:28.999158    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:29.094117 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:29.094090    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:29.192693 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:29.192668    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:32.307307 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.307272    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"]
Jul 13 20:22:32.307694 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.307549    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerName="discovery"
Jul 13 20:22:32.307694 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.307560    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerName="discovery"
Jul 13 20:22:32.307694 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.307599    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="87aa6726-845d-4e69-bbb5-01017c7e8a57" containerName="discovery"
Jul 13 20:22:32.309406 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.309377    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.317013 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.316995    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lf9fh\""
Jul 13 20:22:32.317600 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.317581    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 20:22:32.318433 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.318418    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 20:22:32.318930 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.318914    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 13 20:22:32.320156 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.320141    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 13 20:22:32.321322 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.321297    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:32.327387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.327367    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"]
Jul 13 20:22:32.404652 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.404625    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r5d\" (UniqueName: \"kubernetes.io/projected/fb68c277-61e1-42a4-a595-cd88fc4686fc-kube-api-access-h8r5d\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.404760 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.404714    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb68c277-61e1-42a4-a595-cd88fc4686fc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.404760 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.404751    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb68c277-61e1-42a4-a595-cd88fc4686fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.426490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.426469    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:32.505689 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.505659    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb68c277-61e1-42a4-a595-cd88fc4686fc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.505780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.505696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb68c277-61e1-42a4-a595-cd88fc4686fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.505780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.505718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r5d\" (UniqueName: \"kubernetes.io/projected/fb68c277-61e1-42a4-a595-cd88fc4686fc-kube-api-access-h8r5d\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.506553 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.506425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb68c277-61e1-42a4-a595-cd88fc4686fc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.508276 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.508257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb68c277-61e1-42a4-a595-cd88fc4686fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.516184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.516165    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r5d\" (UniqueName: \"kubernetes.io/projected/fb68c277-61e1-42a4-a595-cd88fc4686fc-kube-api-access-h8r5d\") pod \"kuadrant-console-plugin-66b5774985-jknvw\" (UID: \"fb68c277-61e1-42a4-a595-cd88fc4686fc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.618310 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.618289    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"
Jul 13 20:22:32.746273 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.746248    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw"]
Jul 13 20:22:32.748046 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:22:32.748019    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb68c277_61e1_42a4_a595_cd88fc4686fc.slice/crio-9d353edebae9b007299e16f12db40db6a8c6540159ac13bfdbbadfaffb266e2f WatchSource:0}: Error finding container 9d353edebae9b007299e16f12db40db6a8c6540159ac13bfdbbadfaffb266e2f: Status 404 returned error can't find the container with id 9d353edebae9b007299e16f12db40db6a8c6540159ac13bfdbbadfaffb266e2f
Jul 13 20:22:32.906688 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:32.906609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:33.452644 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:33.452608    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw" event={"ID":"fb68c277-61e1-42a4-a595-cd88fc4686fc","Type":"ContainerStarted","Data":"9d353edebae9b007299e16f12db40db6a8c6540159ac13bfdbbadfaffb266e2f"}
Jul 13 20:22:33.905966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:33.905927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:34.007249 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:34.007212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:40.193310 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:40.193269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:40.293620 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:40.293583    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:45.037070 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:45.037033    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:45.128963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:45.128932    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:48.726545 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:48.726502    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:48.872781 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:48.872747    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:49.116889 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:49.116837    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:49.177057 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:49.177020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:53.116824 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:53.116791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:53.181176 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:53.181149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:55.546826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:55.546784    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw" event={"ID":"fb68c277-61e1-42a4-a595-cd88fc4686fc","Type":"ContainerStarted","Data":"4caf468737ab30000a7054ae32627345b65a060870ec40d870da15a97ee9019f"}
Jul 13 20:22:55.565527 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:22:55.565479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jknvw" podStartSLOduration=1.420177855 podStartE2EDuration="23.565464074s" totalImagesPullingTime="22.145286219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:22:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:22:55.563689879 +0000 UTC m=+509.464392541" watchObservedRunningTime="2026-07-13 20:22:55.565464074 +0000 UTC m=+509.466166735"
Jul 13 20:23:00.527740 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:00.527702    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:00.595812 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:00.595775    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:04.331203 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:04.331164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:04.441127 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:04.441092    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:12.376201 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:12.376165    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:12.657449 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:12.657354    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:17.128979 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:17.128942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:17.269187 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:17.269154    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:17.383471 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:17.383387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:18.015616 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.015586    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-vrsr5"]
Jul 13 20:23:18.048267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.048226    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:18.071406 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.071373    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-vrsr5"]
Jul 13 20:23:18.071506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.071466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:18.074292 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.074264    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2xx7s\""
Jul 13 20:23:18.142633 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.142607    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8kj\" (UniqueName: \"kubernetes.io/projected/da1467a0-2a99-469d-9918-3a436ab8d99e-kube-api-access-vz8kj\") pod \"authorino-58cd955675-vrsr5\" (UID: \"da1467a0-2a99-469d-9918-3a436ab8d99e\") " pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:18.148922 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.148900    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:18.161735 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.161708    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-r8xql"]
Jul 13 20:23:18.194633 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.194612    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r8xql"]
Jul 13 20:23:18.194715 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.194682    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:18.243007 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.242964    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8kj\" (UniqueName: \"kubernetes.io/projected/da1467a0-2a99-469d-9918-3a436ab8d99e-kube-api-access-vz8kj\") pod \"authorino-58cd955675-vrsr5\" (UID: \"da1467a0-2a99-469d-9918-3a436ab8d99e\") " pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:18.243102 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.243078    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99s5\" (UniqueName: \"kubernetes.io/projected/93191570-abcb-47a0-9c33-f4162e72c5fb-kube-api-access-z99s5\") pod \"authorino-75d556bd77-r8xql\" (UID: \"93191570-abcb-47a0-9c33-f4162e72c5fb\") " pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:18.251935 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.251914    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8kj\" (UniqueName: \"kubernetes.io/projected/da1467a0-2a99-469d-9918-3a436ab8d99e-kube-api-access-vz8kj\") pod \"authorino-58cd955675-vrsr5\" (UID: \"da1467a0-2a99-469d-9918-3a436ab8d99e\") " pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:18.343946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.343927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99s5\" (UniqueName: \"kubernetes.io/projected/93191570-abcb-47a0-9c33-f4162e72c5fb-kube-api-access-z99s5\") pod \"authorino-75d556bd77-r8xql\" (UID: \"93191570-abcb-47a0-9c33-f4162e72c5fb\") " pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:18.372010 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.371989    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99s5\" (UniqueName: \"kubernetes.io/projected/93191570-abcb-47a0-9c33-f4162e72c5fb-kube-api-access-z99s5\") pod \"authorino-75d556bd77-r8xql\" (UID: \"93191570-abcb-47a0-9c33-f4162e72c5fb\") " pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:18.380146 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.380129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:18.502731 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.502704    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:18.542178 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.542155    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-vrsr5"]
Jul 13 20:23:18.544610 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:23:18.544572    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda1467a0_2a99_469d_9918_3a436ab8d99e.slice/crio-9d95512df24a575a97d774c9dd8cd892a7797acaab28f8f8be9fc267988bfee8 WatchSource:0}: Error finding container 9d95512df24a575a97d774c9dd8cd892a7797acaab28f8f8be9fc267988bfee8: Status 404 returned error can't find the container with id 9d95512df24a575a97d774c9dd8cd892a7797acaab28f8f8be9fc267988bfee8
Jul 13 20:23:18.625671 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.625642    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vrsr5" event={"ID":"da1467a0-2a99-469d-9918-3a436ab8d99e","Type":"ContainerStarted","Data":"9d95512df24a575a97d774c9dd8cd892a7797acaab28f8f8be9fc267988bfee8"}
Jul 13 20:23:18.680373 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:18.680345    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r8xql"]
Jul 13 20:23:18.683233 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:23:18.683207    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93191570_abcb_47a0_9c33_f4162e72c5fb.slice/crio-767165a0d6965c1136b5e6e520d39bfe4b7ddf1459ee63275eef3c221b7dd67b WatchSource:0}: Error finding container 767165a0d6965c1136b5e6e520d39bfe4b7ddf1459ee63275eef3c221b7dd67b: Status 404 returned error can't find the container with id 767165a0d6965c1136b5e6e520d39bfe4b7ddf1459ee63275eef3c221b7dd67b
Jul 13 20:23:19.643625 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:19.643584    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r8xql" event={"ID":"93191570-abcb-47a0-9c33-f4162e72c5fb","Type":"ContainerStarted","Data":"767165a0d6965c1136b5e6e520d39bfe4b7ddf1459ee63275eef3c221b7dd67b"}
Jul 13 20:23:22.655650 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:22.655613    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vrsr5" event={"ID":"da1467a0-2a99-469d-9918-3a436ab8d99e","Type":"ContainerStarted","Data":"3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0"}
Jul 13 20:23:22.657029 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:22.657005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r8xql" event={"ID":"93191570-abcb-47a0-9c33-f4162e72c5fb","Type":"ContainerStarted","Data":"ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af"}
Jul 13 20:23:22.671053 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:22.671012    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-vrsr5" podStartSLOduration=1.033553597 podStartE2EDuration="4.670998737s" totalImagesPullingTime="3.63744514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:23:22.670345627 +0000 UTC m=+536.571048289" watchObservedRunningTime="2026-07-13 20:23:22.670998737 +0000 UTC m=+536.571701399"
Jul 13 20:23:22.686103 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:22.686055    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-r8xql" podStartSLOduration=1.197131067 podStartE2EDuration="4.686041805s" totalImagesPullingTime="3.488910738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:23:22.685733306 +0000 UTC m=+536.586435967" watchObservedRunningTime="2026-07-13 20:23:22.686041805 +0000 UTC m=+536.586744466"
Jul 13 20:23:22.712444 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:22.712417    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-vrsr5"]
Jul 13 20:23:24.664252 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:24.664213    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-vrsr5" podUID="da1467a0-2a99-469d-9918-3a436ab8d99e" containerName="authorino" containerID="cri-o://3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0" gracePeriod=30
Jul 13 20:23:24.911754 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:24.911730    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:24.995628 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:24.995552    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz8kj\" (UniqueName: \"kubernetes.io/projected/da1467a0-2a99-469d-9918-3a436ab8d99e-kube-api-access-vz8kj\") pod \"da1467a0-2a99-469d-9918-3a436ab8d99e\" (UID: \"da1467a0-2a99-469d-9918-3a436ab8d99e\") "
Jul 13 20:23:24.997814 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:24.997785    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1467a0-2a99-469d-9918-3a436ab8d99e-kube-api-access-vz8kj" (OuterVolumeSpecName: "kube-api-access-vz8kj") pod "da1467a0-2a99-469d-9918-3a436ab8d99e" (UID: "da1467a0-2a99-469d-9918-3a436ab8d99e"). InnerVolumeSpecName "kube-api-access-vz8kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:23:25.096373 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.096336    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz8kj\" (UniqueName: \"kubernetes.io/projected/da1467a0-2a99-469d-9918-3a436ab8d99e-kube-api-access-vz8kj\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:23:25.668806 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.668767    2584 generic.go:356] "Generic (PLEG): container finished" podID="da1467a0-2a99-469d-9918-3a436ab8d99e" containerID="3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0" exitCode=0
Jul 13 20:23:25.668806 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.668807    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-vrsr5"
Jul 13 20:23:25.669278 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.668870    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vrsr5" event={"ID":"da1467a0-2a99-469d-9918-3a436ab8d99e","Type":"ContainerDied","Data":"3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0"}
Jul 13 20:23:25.669278 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.668919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-vrsr5" event={"ID":"da1467a0-2a99-469d-9918-3a436ab8d99e","Type":"ContainerDied","Data":"9d95512df24a575a97d774c9dd8cd892a7797acaab28f8f8be9fc267988bfee8"}
Jul 13 20:23:25.669278 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.668945    2584 scope.go:117] "RemoveContainer" containerID="3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0"
Jul 13 20:23:25.677799 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.677781    2584 scope.go:117] "RemoveContainer" containerID="3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0"
Jul 13 20:23:25.678078 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:23:25.678057    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0\": container with ID starting with 3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0 not found: ID does not exist" containerID="3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0"
Jul 13 20:23:25.678177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.678086    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0"} err="failed to get container status \"3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0\": rpc error: code = NotFound desc = could not find container \"3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0\": container with ID starting with 3f6da5f770f99859e58321a67a3613c7d509060a12e0f5405f8fc33f75a5a7b0 not found: ID does not exist"
Jul 13 20:23:25.694549 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.694516    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-vrsr5"]
Jul 13 20:23:25.695785 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:25.695766    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-vrsr5"]
Jul 13 20:23:26.636539 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:26.636507    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1467a0-2a99-469d-9918-3a436ab8d99e" path="/var/lib/kubelet/pods/da1467a0-2a99-469d-9918-3a436ab8d99e/volumes"
Jul 13 20:23:27.387753 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:27.387698    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:27.440767 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:27.440736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:28.662691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:28.662659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:38.374775 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:38.374742    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:38.473797 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:38.473755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:38.909493 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:38.909457    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:50.554981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.554938    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:50.663437 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.663387    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:50.797198 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.797165    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdcgc"]
Jul 13 20:23:50.797456 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.797443    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1467a0-2a99-469d-9918-3a436ab8d99e" containerName="authorino"
Jul 13 20:23:50.797514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.797459    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1467a0-2a99-469d-9918-3a436ab8d99e" containerName="authorino"
Jul 13 20:23:50.797514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.797511    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1467a0-2a99-469d-9918-3a436ab8d99e" containerName="authorino"
Jul 13 20:23:50.800229 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.800214    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:50.809797 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.809775    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdcgc"]
Jul 13 20:23:50.897112 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.897077    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjfb\" (UniqueName: \"kubernetes.io/projected/6c4a8315-553e-43c4-b2a6-f80d2c05c45c-kube-api-access-9xjfb\") pod \"authorino-5d4b67c446-gdcgc\" (UID: \"6c4a8315-553e-43c4-b2a6-f80d2c05c45c\") " pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:50.997796 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:50.997761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjfb\" (UniqueName: \"kubernetes.io/projected/6c4a8315-553e-43c4-b2a6-f80d2c05c45c-kube-api-access-9xjfb\") pod \"authorino-5d4b67c446-gdcgc\" (UID: \"6c4a8315-553e-43c4-b2a6-f80d2c05c45c\") " pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:51.010573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.010543    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjfb\" (UniqueName: \"kubernetes.io/projected/6c4a8315-553e-43c4-b2a6-f80d2c05c45c-kube-api-access-9xjfb\") pod \"authorino-5d4b67c446-gdcgc\" (UID: \"6c4a8315-553e-43c4-b2a6-f80d2c05c45c\") " pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:51.051873 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.051850    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdcgc"]
Jul 13 20:23:51.052051 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.052040    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:51.079495 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.079469    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c99dd5ffd-jg5kj"]
Jul 13 20:23:51.082700 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.082682    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.085591 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.085571    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 13 20:23:51.092548 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.092528    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c99dd5ffd-jg5kj"]
Jul 13 20:23:51.189464 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.189430    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdcgc"]
Jul 13 20:23:51.192235 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:23:51.192204    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c4a8315_553e_43c4_b2a6_f80d2c05c45c.slice/crio-5c3d04195286a2f170c765053267948bfbede3a3f794c1eaafa2a8ab2c221760 WatchSource:0}: Error finding container 5c3d04195286a2f170c765053267948bfbede3a3f794c1eaafa2a8ab2c221760: Status 404 returned error can't find the container with id 5c3d04195286a2f170c765053267948bfbede3a3f794c1eaafa2a8ab2c221760
Jul 13 20:23:51.199251 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.199231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6w9\" (UniqueName: \"kubernetes.io/projected/41c16406-d6f5-42c9-954a-268b81b51fc8-kube-api-access-xl6w9\") pod \"authorino-6c99dd5ffd-jg5kj\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") " pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.199351 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.199277    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41c16406-d6f5-42c9-954a-268b81b51fc8-tls-cert\") pod \"authorino-6c99dd5ffd-jg5kj\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") " pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.300684 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.300647    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6w9\" (UniqueName: \"kubernetes.io/projected/41c16406-d6f5-42c9-954a-268b81b51fc8-kube-api-access-xl6w9\") pod \"authorino-6c99dd5ffd-jg5kj\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") " pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.300864 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.300712    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41c16406-d6f5-42c9-954a-268b81b51fc8-tls-cert\") pod \"authorino-6c99dd5ffd-jg5kj\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") " pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.303238 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.303219    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41c16406-d6f5-42c9-954a-268b81b51fc8-tls-cert\") pod \"authorino-6c99dd5ffd-jg5kj\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") " pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.310533 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.310506    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6w9\" (UniqueName: \"kubernetes.io/projected/41c16406-d6f5-42c9-954a-268b81b51fc8-kube-api-access-xl6w9\") pod \"authorino-6c99dd5ffd-jg5kj\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") " pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.392311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.392282    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:23:51.520580 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.520550    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c99dd5ffd-jg5kj"]
Jul 13 20:23:51.524019 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:23:51.523996    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41c16406_d6f5_42c9_954a_268b81b51fc8.slice/crio-3707cbebe7429b284d9b60913e002b367a0aa85ff71fb51ed7c2adfcf77aa133 WatchSource:0}: Error finding container 3707cbebe7429b284d9b60913e002b367a0aa85ff71fb51ed7c2adfcf77aa133: Status 404 returned error can't find the container with id 3707cbebe7429b284d9b60913e002b367a0aa85ff71fb51ed7c2adfcf77aa133
Jul 13 20:23:51.760502 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.760405    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj" event={"ID":"41c16406-d6f5-42c9-954a-268b81b51fc8","Type":"ContainerStarted","Data":"3707cbebe7429b284d9b60913e002b367a0aa85ff71fb51ed7c2adfcf77aa133"}
Jul 13 20:23:51.761486 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:51.761465    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdcgc" event={"ID":"6c4a8315-553e-43c4-b2a6-f80d2c05c45c","Type":"ContainerStarted","Data":"5c3d04195286a2f170c765053267948bfbede3a3f794c1eaafa2a8ab2c221760"}
Jul 13 20:23:52.767143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.767102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj" event={"ID":"41c16406-d6f5-42c9-954a-268b81b51fc8","Type":"ContainerStarted","Data":"b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758"}
Jul 13 20:23:52.768621 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.768578    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdcgc" event={"ID":"6c4a8315-553e-43c4-b2a6-f80d2c05c45c","Type":"ContainerStarted","Data":"c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612"}
Jul 13 20:23:52.768737 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.768669    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-gdcgc" podUID="6c4a8315-553e-43c4-b2a6-f80d2c05c45c" containerName="authorino" containerID="cri-o://c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612" gracePeriod=30
Jul 13 20:23:52.785031 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.784988    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj" podStartSLOduration=1.263316548 podStartE2EDuration="1.784970534s" totalImagesPullingTime="521.653986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:23:52.784729281 +0000 UTC m=+566.685431943" watchObservedRunningTime="2026-07-13 20:23:52.784970534 +0000 UTC m=+566.685673197"
Jul 13 20:23:52.800080 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.800033    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-gdcgc" podStartSLOduration=2.108385777 podStartE2EDuration="2.800017946s" totalImagesPullingTime="691.632169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:23:52.799834518 +0000 UTC m=+566.700537182" watchObservedRunningTime="2026-07-13 20:23:52.800017946 +0000 UTC m=+566.700720608"
Jul 13 20:23:52.815901 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.815867    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r8xql"]
Jul 13 20:23:52.816108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:52.816081    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-r8xql" podUID="93191570-abcb-47a0-9c33-f4162e72c5fb" containerName="authorino" containerID="cri-o://ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af" gracePeriod=30
Jul 13 20:23:53.078109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.078088    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:53.080960 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.080942    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:53.217219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.217188    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xjfb\" (UniqueName: \"kubernetes.io/projected/6c4a8315-553e-43c4-b2a6-f80d2c05c45c-kube-api-access-9xjfb\") pod \"6c4a8315-553e-43c4-b2a6-f80d2c05c45c\" (UID: \"6c4a8315-553e-43c4-b2a6-f80d2c05c45c\") "
Jul 13 20:23:53.217387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.217246    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z99s5\" (UniqueName: \"kubernetes.io/projected/93191570-abcb-47a0-9c33-f4162e72c5fb-kube-api-access-z99s5\") pod \"93191570-abcb-47a0-9c33-f4162e72c5fb\" (UID: \"93191570-abcb-47a0-9c33-f4162e72c5fb\") "
Jul 13 20:23:53.219470 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.219435    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4a8315-553e-43c4-b2a6-f80d2c05c45c-kube-api-access-9xjfb" (OuterVolumeSpecName: "kube-api-access-9xjfb") pod "6c4a8315-553e-43c4-b2a6-f80d2c05c45c" (UID: "6c4a8315-553e-43c4-b2a6-f80d2c05c45c"). InnerVolumeSpecName "kube-api-access-9xjfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:23:53.219579 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.219466    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93191570-abcb-47a0-9c33-f4162e72c5fb-kube-api-access-z99s5" (OuterVolumeSpecName: "kube-api-access-z99s5") pod "93191570-abcb-47a0-9c33-f4162e72c5fb" (UID: "93191570-abcb-47a0-9c33-f4162e72c5fb"). InnerVolumeSpecName "kube-api-access-z99s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:23:53.318688 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.318614    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjfb\" (UniqueName: \"kubernetes.io/projected/6c4a8315-553e-43c4-b2a6-f80d2c05c45c-kube-api-access-9xjfb\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:23:53.318688 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.318644    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99s5\" (UniqueName: \"kubernetes.io/projected/93191570-abcb-47a0-9c33-f4162e72c5fb-kube-api-access-z99s5\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:23:53.773813 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.773778    2584 generic.go:356] "Generic (PLEG): container finished" podID="93191570-abcb-47a0-9c33-f4162e72c5fb" containerID="ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af" exitCode=0
Jul 13 20:23:53.774267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.773836    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-r8xql"
Jul 13 20:23:53.774267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.773860    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r8xql" event={"ID":"93191570-abcb-47a0-9c33-f4162e72c5fb","Type":"ContainerDied","Data":"ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af"}
Jul 13 20:23:53.774267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.773900    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-r8xql" event={"ID":"93191570-abcb-47a0-9c33-f4162e72c5fb","Type":"ContainerDied","Data":"767165a0d6965c1136b5e6e520d39bfe4b7ddf1459ee63275eef3c221b7dd67b"}
Jul 13 20:23:53.774267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.773916    2584 scope.go:117] "RemoveContainer" containerID="ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af"
Jul 13 20:23:53.775223 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.775204    2584 generic.go:356] "Generic (PLEG): container finished" podID="6c4a8315-553e-43c4-b2a6-f80d2c05c45c" containerID="c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612" exitCode=0
Jul 13 20:23:53.775518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.775269    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdcgc"
Jul 13 20:23:53.775518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.775313    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdcgc" event={"ID":"6c4a8315-553e-43c4-b2a6-f80d2c05c45c","Type":"ContainerDied","Data":"c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612"}
Jul 13 20:23:53.775518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.775340    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdcgc" event={"ID":"6c4a8315-553e-43c4-b2a6-f80d2c05c45c","Type":"ContainerDied","Data":"5c3d04195286a2f170c765053267948bfbede3a3f794c1eaafa2a8ab2c221760"}
Jul 13 20:23:53.782061 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.782044    2584 scope.go:117] "RemoveContainer" containerID="ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af"
Jul 13 20:23:53.782348 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:23:53.782325    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af\": container with ID starting with ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af not found: ID does not exist" containerID="ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af"
Jul 13 20:23:53.782437 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.782362    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af"} err="failed to get container status \"ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af\": rpc error: code = NotFound desc = could not find container \"ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af\": container with ID starting with ef8fe339ae66d1a0e9080fb490c181ab83ac085e0db35b920cdc204b18f0d0af not found: ID does not exist"
Jul 13 20:23:53.782437 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.782379    2584 scope.go:117] "RemoveContainer" containerID="c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612"
Jul 13 20:23:53.789676 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.789656    2584 scope.go:117] "RemoveContainer" containerID="c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612"
Jul 13 20:23:53.789922 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:23:53.789903    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612\": container with ID starting with c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612 not found: ID does not exist" containerID="c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612"
Jul 13 20:23:53.789976 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.789930    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612"} err="failed to get container status \"c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612\": rpc error: code = NotFound desc = could not find container \"c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612\": container with ID starting with c541ac9dc56290b249c32eea9be16d8baa825655e93aadb551f2b2cd29bfc612 not found: ID does not exist"
Jul 13 20:23:53.799135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.799114    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdcgc"]
Jul 13 20:23:53.803199 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.803177    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdcgc"]
Jul 13 20:23:53.814535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.814512    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r8xql"]
Jul 13 20:23:53.818237 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:53.818217    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-r8xql"]
Jul 13 20:23:54.636946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:54.636914    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4a8315-553e-43c4-b2a6-f80d2c05c45c" path="/var/lib/kubelet/pods/6c4a8315-553e-43c4-b2a6-f80d2c05c45c/volumes"
Jul 13 20:23:54.637212 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:54.637200    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93191570-abcb-47a0-9c33-f4162e72c5fb" path="/var/lib/kubelet/pods/93191570-abcb-47a0-9c33-f4162e72c5fb/volumes"
Jul 13 20:23:57.211104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:57.211070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:57.307871 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:57.307830    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:57.400012 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:23:57.399977    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:00.215219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:00.215179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:00.378120 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:00.378070    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:29.840711 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:29.840682    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:29.902118 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:29.902082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:31.533246 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:31.533195    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:31.632167 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:31.632124    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:31.965074 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:31.965040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:32.094417 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:32.094370    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:32.574115 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:32.574084    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:32.679000 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:32.678973    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:42.987913 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:42.987876    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:44.505475 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:24:44.505437    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:12.819539 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:12.819496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:17.507428 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507379    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"]
Jul 13 20:25:17.507826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507651    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4a8315-553e-43c4-b2a6-f80d2c05c45c" containerName="authorino"
Jul 13 20:25:17.507826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507661    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4a8315-553e-43c4-b2a6-f80d2c05c45c" containerName="authorino"
Jul 13 20:25:17.507826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507673    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93191570-abcb-47a0-9c33-f4162e72c5fb" containerName="authorino"
Jul 13 20:25:17.507826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507678    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="93191570-abcb-47a0-9c33-f4162e72c5fb" containerName="authorino"
Jul 13 20:25:17.507826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507727    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="93191570-abcb-47a0-9c33-f4162e72c5fb" containerName="authorino"
Jul 13 20:25:17.507826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.507736    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4a8315-553e-43c4-b2a6-f80d2c05c45c" containerName="authorino"
Jul 13 20:25:17.510519 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.510503    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.514576 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.514547    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wnn2k\""
Jul 13 20:25:17.514718 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.514578    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 13 20:25:17.514718 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.514593    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 13 20:25:17.514718 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.514612    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 13 20:25:17.523135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.523117    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"]
Jul 13 20:25:17.592628 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.592591    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:17.630599 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.630576    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.630738 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.630607    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.630738 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.630627    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0067c0d-2a28-4c0b-9c52-b354b794900b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.630738 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.630654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j44f6\" (UniqueName: \"kubernetes.io/projected/b0067c0d-2a28-4c0b-9c52-b354b794900b-kube-api-access-j44f6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.630738 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.630705    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.630895 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.630790    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.731814 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.731781    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.731814 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.731817    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732025 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.731844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0067c0d-2a28-4c0b-9c52-b354b794900b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732025 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.731875    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j44f6\" (UniqueName: \"kubernetes.io/projected/b0067c0d-2a28-4c0b-9c52-b354b794900b-kube-api-access-j44f6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732025 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.731892    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732025 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.731913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732231 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.732204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732295 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.732247    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.732348 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.732314    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.734318 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.734292    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0067c0d-2a28-4c0b-9c52-b354b794900b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.734510 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.734474    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0067c0d-2a28-4c0b-9c52-b354b794900b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.741133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.741103    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j44f6\" (UniqueName: \"kubernetes.io/projected/b0067c0d-2a28-4c0b-9c52-b354b794900b-kube-api-access-j44f6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wdg89\" (UID: \"b0067c0d-2a28-4c0b-9c52-b354b794900b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.819752 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.819663    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:17.948429 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:17.948380    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"]
Jul 13 20:25:17.951058 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:25:17.951035    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0067c0d_2a28_4c0b_9c52_b354b794900b.slice/crio-7620d980d3d3991a0af735dc48ca29c54dd9c291597b0b9f480a26d7bd0425f7 WatchSource:0}: Error finding container 7620d980d3d3991a0af735dc48ca29c54dd9c291597b0b9f480a26d7bd0425f7: Status 404 returned error can't find the container with id 7620d980d3d3991a0af735dc48ca29c54dd9c291597b0b9f480a26d7bd0425f7
Jul 13 20:25:18.085238 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:18.085146    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89" event={"ID":"b0067c0d-2a28-4c0b-9c52-b354b794900b","Type":"ContainerStarted","Data":"7620d980d3d3991a0af735dc48ca29c54dd9c291597b0b9f480a26d7bd0425f7"}
Jul 13 20:25:19.090517 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:19.090487    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89" event={"ID":"b0067c0d-2a28-4c0b-9c52-b354b794900b","Type":"ContainerStarted","Data":"8e54403fad6706d7edc96ce7683526a8d87e0ee206f051289afb2d39f2b8a0c7"}
Jul 13 20:25:19.110115 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:19.109823    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89" podStartSLOduration=1.031910316 podStartE2EDuration="2.10980133s" totalImagesPullingTime="1.077891014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:25:19.108992801 +0000 UTC m=+653.009695462" watchObservedRunningTime="2026-07-13 20:25:19.10980133 +0000 UTC m=+653.010503992"
Jul 13 20:25:21.924125 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:21.924056    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:26.915411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:26.915366    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:27.820059 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:27.820023    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:27.820238 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:27.820102    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:27.832320 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:27.832295    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:28.139976 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:28.139946    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wdg89"
Jul 13 20:25:42.728746 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:42.728707    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:52.728068 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:52.728028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:58.115122 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.115089    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"]
Jul 13 20:25:58.118428 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.118385    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.122265 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.122237    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 13 20:25:58.134890 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.134869    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.134978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.134896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9225f2a5-8be3-475c-aadf-5c44a1b26247-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.134978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.134927    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.134978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.134951    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.135104 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.135032    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.135140 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.135102    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhm7\" (UniqueName: \"kubernetes.io/projected/9225f2a5-8be3-475c-aadf-5c44a1b26247-kube-api-access-bvhm7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.143445 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.143424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"]
Jul 13 20:25:58.180653 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.180609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:58.235680 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235656    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.235762 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235687    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.235762 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235715    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.235857 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235763    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhm7\" (UniqueName: \"kubernetes.io/projected/9225f2a5-8be3-475c-aadf-5c44a1b26247-kube-api-access-bvhm7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.235912 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.235958 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9225f2a5-8be3-475c-aadf-5c44a1b26247-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.236010 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.235986    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.236223 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.236204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.236294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.236261    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.238107 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.238085    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9225f2a5-8be3-475c-aadf-5c44a1b26247-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.238350 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.238333    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9225f2a5-8be3-475c-aadf-5c44a1b26247-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.246423 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.246376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhm7\" (UniqueName: \"kubernetes.io/projected/9225f2a5-8be3-475c-aadf-5c44a1b26247-kube-api-access-bvhm7\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s\" (UID: \"9225f2a5-8be3-475c-aadf-5c44a1b26247\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.426911 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.426849    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:25:58.554717 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.554689    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"]
Jul 13 20:25:58.557047 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:25:58.557022    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9225f2a5_8be3_475c_aadf_5c44a1b26247.slice/crio-ca752f56d865a05e635ab0419e51608f77292ad488cc10da345f693f23ec1ba6 WatchSource:0}: Error finding container ca752f56d865a05e635ab0419e51608f77292ad488cc10da345f693f23ec1ba6: Status 404 returned error can't find the container with id ca752f56d865a05e635ab0419e51608f77292ad488cc10da345f693f23ec1ba6
Jul 13 20:25:58.558369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:58.558349    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:25:59.248771 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:59.248689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s" event={"ID":"9225f2a5-8be3-475c-aadf-5c44a1b26247","Type":"ContainerStarted","Data":"b807e3ac32d094c5dff2f5cfa96469c98b38d19912503fecbdc6132c0dfe690f"}
Jul 13 20:25:59.248771 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:59.248733    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s" event={"ID":"9225f2a5-8be3-475c-aadf-5c44a1b26247","Type":"ContainerStarted","Data":"ca752f56d865a05e635ab0419e51608f77292ad488cc10da345f693f23ec1ba6"}
Jul 13 20:25:59.267281 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:25:59.267221    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s" podStartSLOduration=0.955031153 podStartE2EDuration="1.267203598s" totalImagesPullingTime="312.172445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:25:59.266370906 +0000 UTC m=+693.167073568" watchObservedRunningTime="2026-07-13 20:25:59.267203598 +0000 UTC m=+693.167906260"
Jul 13 20:26:08.427013 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:08.426976    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:26:08.427458 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:08.427282    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:26:08.439664 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:08.439634    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:26:09.300826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:09.300800    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s"
Jul 13 20:26:39.898070 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:39.898031    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:50.829586 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:50.829542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:50.888535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:50.888496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:57.434144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.434108    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c9d6b7c9f-rbx8q"]
Jul 13 20:26:57.436489 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.436472    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.445415 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.445373    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c9d6b7c9f-rbx8q"]
Jul 13 20:26:57.497700 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.497657    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7176977e-78c3-4e15-a0db-d2b754cc6994-tls-cert\") pod \"authorino-c9d6b7c9f-rbx8q\" (UID: \"7176977e-78c3-4e15-a0db-d2b754cc6994\") " pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.497877 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.497727    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmhm9\" (UniqueName: \"kubernetes.io/projected/7176977e-78c3-4e15-a0db-d2b754cc6994-kube-api-access-bmhm9\") pod \"authorino-c9d6b7c9f-rbx8q\" (UID: \"7176977e-78c3-4e15-a0db-d2b754cc6994\") " pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.598348 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.598296    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7176977e-78c3-4e15-a0db-d2b754cc6994-tls-cert\") pod \"authorino-c9d6b7c9f-rbx8q\" (UID: \"7176977e-78c3-4e15-a0db-d2b754cc6994\") " pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.598510 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.598368    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmhm9\" (UniqueName: \"kubernetes.io/projected/7176977e-78c3-4e15-a0db-d2b754cc6994-kube-api-access-bmhm9\") pod \"authorino-c9d6b7c9f-rbx8q\" (UID: \"7176977e-78c3-4e15-a0db-d2b754cc6994\") " pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.600969 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.600947    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7176977e-78c3-4e15-a0db-d2b754cc6994-tls-cert\") pod \"authorino-c9d6b7c9f-rbx8q\" (UID: \"7176977e-78c3-4e15-a0db-d2b754cc6994\") " pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.606707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.606683    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmhm9\" (UniqueName: \"kubernetes.io/projected/7176977e-78c3-4e15-a0db-d2b754cc6994-kube-api-access-bmhm9\") pod \"authorino-c9d6b7c9f-rbx8q\" (UID: \"7176977e-78c3-4e15-a0db-d2b754cc6994\") " pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.672132 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.672098    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:57.746160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.746075    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q"
Jul 13 20:26:57.769117 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.769085    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:57.879989 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:57.879965    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c9d6b7c9f-rbx8q"]
Jul 13 20:26:57.882352 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:26:57.882325    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7176977e_78c3_4e15_a0db_d2b754cc6994.slice/crio-25fe094c7cf4c55067ba91ccb08227dd5185c46c91aa3ecada30e25c81421fd5 WatchSource:0}: Error finding container 25fe094c7cf4c55067ba91ccb08227dd5185c46c91aa3ecada30e25c81421fd5: Status 404 returned error can't find the container with id 25fe094c7cf4c55067ba91ccb08227dd5185c46c91aa3ecada30e25c81421fd5
Jul 13 20:26:58.480703 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.480666    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q" event={"ID":"7176977e-78c3-4e15-a0db-d2b754cc6994","Type":"ContainerStarted","Data":"45fe54d9db957159a05d2cd2577d8a60c38fd9234107f77fe31b7be41d6b49ba"}
Jul 13 20:26:58.481066 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.480711    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q" event={"ID":"7176977e-78c3-4e15-a0db-d2b754cc6994","Type":"ContainerStarted","Data":"25fe094c7cf4c55067ba91ccb08227dd5185c46c91aa3ecada30e25c81421fd5"}
Jul 13 20:26:58.504754 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.504691    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c9d6b7c9f-rbx8q" podStartSLOduration=1.055115457 podStartE2EDuration="1.504669082s" totalImagesPullingTime="449.553625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:26:58.503302725 +0000 UTC m=+752.404005386" watchObservedRunningTime="2026-07-13 20:26:58.504669082 +0000 UTC m=+752.405371744"
Jul 13 20:26:58.545651 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.545618    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c99dd5ffd-jg5kj"]
Jul 13 20:26:58.545847 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.545786    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj" podUID="41c16406-d6f5-42c9-954a-268b81b51fc8" containerName="authorino" containerID="cri-o://b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758" gracePeriod=30
Jul 13 20:26:58.795995 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.795955    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:26:58.807486 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.807457    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41c16406-d6f5-42c9-954a-268b81b51fc8-tls-cert\") pod \"41c16406-d6f5-42c9-954a-268b81b51fc8\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") "
Jul 13 20:26:58.807689 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.807498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl6w9\" (UniqueName: \"kubernetes.io/projected/41c16406-d6f5-42c9-954a-268b81b51fc8-kube-api-access-xl6w9\") pod \"41c16406-d6f5-42c9-954a-268b81b51fc8\" (UID: \"41c16406-d6f5-42c9-954a-268b81b51fc8\") "
Jul 13 20:26:58.809785 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.809755    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c16406-d6f5-42c9-954a-268b81b51fc8-kube-api-access-xl6w9" (OuterVolumeSpecName: "kube-api-access-xl6w9") pod "41c16406-d6f5-42c9-954a-268b81b51fc8" (UID: "41c16406-d6f5-42c9-954a-268b81b51fc8"). InnerVolumeSpecName "kube-api-access-xl6w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:26:58.819771 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.819720    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c16406-d6f5-42c9-954a-268b81b51fc8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "41c16406-d6f5-42c9-954a-268b81b51fc8" (UID: "41c16406-d6f5-42c9-954a-268b81b51fc8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:26:58.908773 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.908697    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6w9\" (UniqueName: \"kubernetes.io/projected/41c16406-d6f5-42c9-954a-268b81b51fc8-kube-api-access-xl6w9\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:26:58.908773 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.908720    2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41c16406-d6f5-42c9-954a-268b81b51fc8-tls-cert\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:26:58.996049 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:58.996006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:59.092064 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.092025    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:59.487841 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.487804    2584 generic.go:356] "Generic (PLEG): container finished" podID="41c16406-d6f5-42c9-954a-268b81b51fc8" containerID="b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758" exitCode=0
Jul 13 20:26:59.488296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.487863    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj"
Jul 13 20:26:59.488296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.487895    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj" event={"ID":"41c16406-d6f5-42c9-954a-268b81b51fc8","Type":"ContainerDied","Data":"b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758"}
Jul 13 20:26:59.488296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.487942    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c99dd5ffd-jg5kj" event={"ID":"41c16406-d6f5-42c9-954a-268b81b51fc8","Type":"ContainerDied","Data":"3707cbebe7429b284d9b60913e002b367a0aa85ff71fb51ed7c2adfcf77aa133"}
Jul 13 20:26:59.488296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.487965    2584 scope.go:117] "RemoveContainer" containerID="b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758"
Jul 13 20:26:59.497534 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.497513    2584 scope.go:117] "RemoveContainer" containerID="b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758"
Jul 13 20:26:59.497983 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:26:59.497853    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758\": container with ID starting with b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758 not found: ID does not exist" containerID="b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758"
Jul 13 20:26:59.497983 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.497889    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758"} err="failed to get container status \"b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758\": rpc error: code = NotFound desc = could not find container \"b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758\": container with ID starting with b66b6834c9bc1a54a11550aa51c3720da30f08a3f9a00bcc7797e39a06c5c758 not found: ID does not exist"
Jul 13 20:26:59.546064 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.546034    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c99dd5ffd-jg5kj"]
Jul 13 20:26:59.562599 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:26:59.562560    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c99dd5ffd-jg5kj"]
Jul 13 20:27:00.638824 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:27:00.638791    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c16406-d6f5-42c9-954a-268b81b51fc8" path="/var/lib/kubelet/pods/41c16406-d6f5-42c9-954a-268b81b51fc8/volumes"
Jul 13 20:27:36.106964 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:27:36.106926    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:10.050919 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:10.050815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:11.318111 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:11.318065    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:17.418466 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:17.418429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:52.177226 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:52.177186    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:52.308287 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:52.308235    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:58.309632 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:58.309595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:58.529108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:58.529064    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:58.668636 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:58.668600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:58.765944 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:28:58.765907    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:29:04.567199 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:29:04.567145    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:29:04.667093 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:29:04.667055    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:00.150762 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.150724    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"]
Jul 13 20:30:00.153133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.151091    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c16406-d6f5-42c9-954a-268b81b51fc8" containerName="authorino"
Jul 13 20:30:00.153133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.151110    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c16406-d6f5-42c9-954a-268b81b51fc8" containerName="authorino"
Jul 13 20:30:00.153133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.151200    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c16406-d6f5-42c9-954a-268b81b51fc8" containerName="authorino"
Jul 13 20:30:00.153993 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.153979    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"
Jul 13 20:30:00.156524 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.156502    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:30:00.163043 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.163027    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"
Jul 13 20:30:00.168263 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.168241    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"]
Jul 13 20:30:00.297360 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:00.297329    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"]
Jul 13 20:30:00.300273 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:30:00.300246    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6866738a_b5f9_4a8f_af63_4121e155309c.slice/crio-0ef20a2f44eb7bd8932b19c53fe70cbdbbd14a1820e9fc58382b71cbd332b60e WatchSource:0}: Error finding container 0ef20a2f44eb7bd8932b19c53fe70cbdbbd14a1820e9fc58382b71cbd332b60e: Status 404 returned error can't find the container with id 0ef20a2f44eb7bd8932b19c53fe70cbdbbd14a1820e9fc58382b71cbd332b60e
Jul 13 20:30:01.199304 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:01.199252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t" event={"ID":"6866738a-b5f9-4a8f-af63-4121e155309c","Type":"ContainerStarted","Data":"0ef20a2f44eb7bd8932b19c53fe70cbdbbd14a1820e9fc58382b71cbd332b60e"}
Jul 13 20:30:03.214018 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:03.213985    2584 generic.go:356] "Generic (PLEG): container finished" podID="6866738a-b5f9-4a8f-af63-4121e155309c" containerID="2ba4c1acb0f76a86d4ddef59196440d0736a52ca8376c8a9bc630c229c83804e" exitCode=0
Jul 13 20:30:03.214330 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:03.214022    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t" event={"ID":"6866738a-b5f9-4a8f-af63-4121e155309c","Type":"ContainerDied","Data":"2ba4c1acb0f76a86d4ddef59196440d0736a52ca8376c8a9bc630c229c83804e"}
Jul 13 20:30:04.344296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:04.344272    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"
Jul 13 20:30:05.223591 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:05.223565    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t"
Jul 13 20:30:05.223768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:05.223559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732910-2pd7t" event={"ID":"6866738a-b5f9-4a8f-af63-4121e155309c","Type":"ContainerDied","Data":"0ef20a2f44eb7bd8932b19c53fe70cbdbbd14a1820e9fc58382b71cbd332b60e"}
Jul 13 20:30:05.223768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:05.223687    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef20a2f44eb7bd8932b19c53fe70cbdbbd14a1820e9fc58382b71cbd332b60e"
Jul 13 20:30:09.100254 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:30:09.100217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:02.599098 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:31:02.598989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:32:46.788705 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:32:46.788669    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:32:46.851283 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:32:46.851247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:04.098543 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:33:04.098506    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:34:57.979212 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:34:57.979172    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:34:58.042943 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:34:58.042896    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:37:01.149516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:37:01.149443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:39:48.191933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:39:48.191842    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:39:52.839229 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:39:52.839185    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:01.223023 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:01.222982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:06.233626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:06.233566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:09.191133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:09.191088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:09.286131 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:09.286090    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:09.634776 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:09.634744    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:16.591605 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:16.591569    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:16.729849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:16.729815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:17.850578 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.850540    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:17.922865 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.922838    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"]
Jul 13 20:40:17.923102 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.923091    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6866738a-b5f9-4a8f-af63-4121e155309c" containerName="cleanup"
Jul 13 20:40:17.923144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.923105    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6866738a-b5f9-4a8f-af63-4121e155309c" containerName="cleanup"
Jul 13 20:40:17.923182 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.923158    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6866738a-b5f9-4a8f-af63-4121e155309c" containerName="cleanup"
Jul 13 20:40:17.925845 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.925828    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:17.928579 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.928561    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b435b208-serving-cert\""
Jul 13 20:40:17.928679 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.928621    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:40:17.929768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.929750    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:40:17.929828 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.929790    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:40:17.938296 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.938272    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"]
Jul 13 20:40:17.944564 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:17.944541    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:18.029638 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.029598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:18.081741 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.081713    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbz7h\" (UniqueName: \"kubernetes.io/projected/6781be21-8257-4581-9733-522ad945880a-kube-api-access-nbz7h\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.081869 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.081771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6781be21-8257-4581-9733-522ad945880a-openshift-service-ca\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.081869 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.081794    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6781be21-8257-4581-9733-522ad945880a-maas-api-tls\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.148012 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.147944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:18.182550 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.182527    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbz7h\" (UniqueName: \"kubernetes.io/projected/6781be21-8257-4581-9733-522ad945880a-kube-api-access-nbz7h\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.182644 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.182560    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6781be21-8257-4581-9733-522ad945880a-openshift-service-ca\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.182644 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.182585    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6781be21-8257-4581-9733-522ad945880a-maas-api-tls\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.183321 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.183304    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6781be21-8257-4581-9733-522ad945880a-openshift-service-ca\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.185171 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.185156    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6781be21-8257-4581-9733-522ad945880a-maas-api-tls\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.192041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.192024    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbz7h\" (UniqueName: \"kubernetes.io/projected/6781be21-8257-4581-9733-522ad945880a-kube-api-access-nbz7h\") pod \"maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.235889 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.235865    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:18.365220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.365194    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"]
Jul 13 20:40:18.367686 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:40:18.367659    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6781be21_8257_4581_9733_522ad945880a.slice/crio-92ec9434ed5f35335dcac906818f68401d11c1395feaf8d5ae41168cdb8a509e WatchSource:0}: Error finding container 92ec9434ed5f35335dcac906818f68401d11c1395feaf8d5ae41168cdb8a509e: Status 404 returned error can't find the container with id 92ec9434ed5f35335dcac906818f68401d11c1395feaf8d5ae41168cdb8a509e
Jul 13 20:40:18.369102 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.369085    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:40:18.661897 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.661869    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6" event={"ID":"6781be21-8257-4581-9733-522ad945880a","Type":"ContainerStarted","Data":"92ec9434ed5f35335dcac906818f68401d11c1395feaf8d5ae41168cdb8a509e"}
Jul 13 20:40:18.883294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.883260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:18.973559 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:18.973480    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:20.408460 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:20.408416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:21.676036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:21.676000    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6" event={"ID":"6781be21-8257-4581-9733-522ad945880a","Type":"ContainerStarted","Data":"760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306"}
Jul 13 20:40:21.676511 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:21.676137    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:21.695142 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:21.695098    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6" podStartSLOduration=2.364089244 podStartE2EDuration="4.695082088s" totalImagesPullingTime="2.330992844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:21.693774851 +0000 UTC m=+1555.594477513" watchObservedRunningTime="2026-07-13 20:40:21.695082088 +0000 UTC m=+1555.595784753"
Jul 13 20:40:27.687512 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:27.687484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:33.420677 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.420646    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"]
Jul 13 20:40:33.421145 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.420886    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6" podUID="6781be21-8257-4581-9733-522ad945880a" containerName="maas-api" containerID="cri-o://760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306" gracePeriod=30
Jul 13 20:40:33.461554 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.461520    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:33.662777 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.662755    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:33.722268 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.722201    2584 generic.go:356] "Generic (PLEG): container finished" podID="6781be21-8257-4581-9733-522ad945880a" containerID="760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306" exitCode=0
Jul 13 20:40:33.722268 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.722242    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"
Jul 13 20:40:33.722426 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.722270    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6" event={"ID":"6781be21-8257-4581-9733-522ad945880a","Type":"ContainerDied","Data":"760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306"}
Jul 13 20:40:33.722426 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.722308    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6" event={"ID":"6781be21-8257-4581-9733-522ad945880a","Type":"ContainerDied","Data":"92ec9434ed5f35335dcac906818f68401d11c1395feaf8d5ae41168cdb8a509e"}
Jul 13 20:40:33.722426 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.722329    2584 scope.go:117] "RemoveContainer" containerID="760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306"
Jul 13 20:40:33.730069 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.730048    2584 scope.go:117] "RemoveContainer" containerID="760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306"
Jul 13 20:40:33.730317 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:40:33.730298    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306\": container with ID starting with 760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306 not found: ID does not exist" containerID="760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306"
Jul 13 20:40:33.730376 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.730326    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306"} err="failed to get container status \"760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306\": rpc error: code = NotFound desc = could not find container \"760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306\": container with ID starting with 760ad5854c9cc471c96b64bd29aa919a9452c04cb60751b9bc11f43a78e08306 not found: ID does not exist"
Jul 13 20:40:33.796981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.796950    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6781be21-8257-4581-9733-522ad945880a-openshift-service-ca\") pod \"6781be21-8257-4581-9733-522ad945880a\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") "
Jul 13 20:40:33.797121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.797025    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbz7h\" (UniqueName: \"kubernetes.io/projected/6781be21-8257-4581-9733-522ad945880a-kube-api-access-nbz7h\") pod \"6781be21-8257-4581-9733-522ad945880a\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") "
Jul 13 20:40:33.797121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.797050    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6781be21-8257-4581-9733-522ad945880a-maas-api-tls\") pod \"6781be21-8257-4581-9733-522ad945880a\" (UID: \"6781be21-8257-4581-9733-522ad945880a\") "
Jul 13 20:40:33.797320 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.797298    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6781be21-8257-4581-9733-522ad945880a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6781be21-8257-4581-9733-522ad945880a" (UID: "6781be21-8257-4581-9733-522ad945880a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:33.799144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.799116    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6781be21-8257-4581-9733-522ad945880a-kube-api-access-nbz7h" (OuterVolumeSpecName: "kube-api-access-nbz7h") pod "6781be21-8257-4581-9733-522ad945880a" (UID: "6781be21-8257-4581-9733-522ad945880a"). InnerVolumeSpecName "kube-api-access-nbz7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:33.799144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.799123    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6781be21-8257-4581-9733-522ad945880a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6781be21-8257-4581-9733-522ad945880a" (UID: "6781be21-8257-4581-9733-522ad945880a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:33.898096 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.898071    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6781be21-8257-4581-9733-522ad945880a-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:40:33.898096 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.898092    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbz7h\" (UniqueName: \"kubernetes.io/projected/6781be21-8257-4581-9733-522ad945880a-kube-api-access-nbz7h\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:40:33.898238 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:33.898102    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6781be21-8257-4581-9733-522ad945880a-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:40:34.044947 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:34.044924    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"]
Jul 13 20:40:34.048926 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:34.048906    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b435b208-fcbcb6545-j5zj6"]
Jul 13 20:40:34.637219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:34.637188    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6781be21-8257-4581-9733-522ad945880a" path="/var/lib/kubelet/pods/6781be21-8257-4581-9733-522ad945880a/volumes"
Jul 13 20:40:37.143769 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:37.143729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:37.964740 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:37.964703    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:44.320980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:44.320940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:44.422469 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:44.422439    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:48.519213 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:48.519179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:48.619298 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:48.619266    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:49.797222 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:49.797189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:49.888020 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:49.887987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:50.007642 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.007603    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"]
Jul 13 20:40:50.008002 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.007988    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6781be21-8257-4581-9733-522ad945880a" containerName="maas-api"
Jul 13 20:40:50.008046 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.008005    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6781be21-8257-4581-9733-522ad945880a" containerName="maas-api"
Jul 13 20:40:50.008085 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.008075    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6781be21-8257-4581-9733-522ad945880a" containerName="maas-api"
Jul 13 20:40:50.012604 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.012578    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.016002 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.015979    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9ebcdac6-serving-cert\""
Jul 13 20:40:50.016119 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.016046    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:40:50.016119 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.016062    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:40:50.017309 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.017293    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:40:50.020835 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.020814    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"]
Jul 13 20:40:50.053651 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.053581    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:50.113804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.113781    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adeaf95f-0d61-449c-b12f-57940be23835-openshift-service-ca\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.113908 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.113818    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.113958 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.113914    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtkr\" (UniqueName: \"kubernetes.io/projected/adeaf95f-0d61-449c-b12f-57940be23835-kube-api-access-5jtkr\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.214796 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.214772    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtkr\" (UniqueName: \"kubernetes.io/projected/adeaf95f-0d61-449c-b12f-57940be23835-kube-api-access-5jtkr\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.214887 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.214806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adeaf95f-0d61-449c-b12f-57940be23835-openshift-service-ca\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.214887 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.214832    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.214960 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:40:50.214950    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-serving-cert: secret "maas-api-e2e-ait-9ebcdac6-serving-cert" not found
Jul 13 20:40:50.215022 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:40:50.215012    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls podName:adeaf95f-0d61-449c-b12f-57940be23835 nodeName:}" failed. No retries permitted until 2026-07-13 20:40:50.71499182 +0000 UTC m=+1584.615694459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls") pod "maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" (UID: "adeaf95f-0d61-449c-b12f-57940be23835") : secret "maas-api-e2e-ait-9ebcdac6-serving-cert" not found
Jul 13 20:40:50.215477 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.215460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adeaf95f-0d61-449c-b12f-57940be23835-openshift-service-ca\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.227823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.227792    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtkr\" (UniqueName: \"kubernetes.io/projected/adeaf95f-0d61-449c-b12f-57940be23835-kube-api-access-5jtkr\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.392449 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.392416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:50.637624 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.637592    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:50.718691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.718620    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.721063 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.721044    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls\") pod \"maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:50.737745 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.737713    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:50.922956 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:50.922923    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:51.065702 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:51.065672    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"]
Jul 13 20:40:51.068652 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:40:51.068626    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadeaf95f_0d61_449c_b12f_57940be23835.slice/crio-8743c0a6a6866ea4e75061d67df5117131534c6c381e25264a398ed673bd1c47 WatchSource:0}: Error finding container 8743c0a6a6866ea4e75061d67df5117131534c6c381e25264a398ed673bd1c47: Status 404 returned error can't find the container with id 8743c0a6a6866ea4e75061d67df5117131534c6c381e25264a398ed673bd1c47
Jul 13 20:40:51.791388 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:51.791348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" event={"ID":"adeaf95f-0d61-449c-b12f-57940be23835","Type":"ContainerStarted","Data":"c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400"}
Jul 13 20:40:51.791594 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:51.791426    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" event={"ID":"adeaf95f-0d61-449c-b12f-57940be23835","Type":"ContainerStarted","Data":"8743c0a6a6866ea4e75061d67df5117131534c6c381e25264a398ed673bd1c47"}
Jul 13 20:40:51.791594 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:51.791454    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:51.811512 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:51.811469    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" podStartSLOduration=2.25416996 podStartE2EDuration="2.811454429s" totalImagesPullingTime="557.284469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:51.810021428 +0000 UTC m=+1585.710724090" watchObservedRunningTime="2026-07-13 20:40:51.811454429 +0000 UTC m=+1585.712157091"
Jul 13 20:40:54.691166 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.691132    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"]
Jul 13 20:40:54.695198 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.695182    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:40:54.697963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.697946    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:40:54.702951 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.702926    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"]
Jul 13 20:40:54.744901 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.744879    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/898516a6-29a6-4444-a8bd-902c4020c57f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v\" (UID: \"898516a6-29a6-4444-a8bd-902c4020c57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:40:54.845555 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.845523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/898516a6-29a6-4444-a8bd-902c4020c57f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v\" (UID: \"898516a6-29a6-4444-a8bd-902c4020c57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:40:54.846144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:54.846123    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/898516a6-29a6-4444-a8bd-902c4020c57f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v\" (UID: \"898516a6-29a6-4444-a8bd-902c4020c57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:40:55.003423 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.003376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:55.005171 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.005152    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:40:55.072920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.072761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:55.134604 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.134583    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"]
Jul 13 20:40:55.137164 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:40:55.137137    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898516a6_29a6_4444_a8bd_902c4020c57f.slice/crio-e1128d370bf75e794b3140153345832b67a0f42a2f50a697c11b417d1532c285 WatchSource:0}: Error finding container e1128d370bf75e794b3140153345832b67a0f42a2f50a697c11b417d1532c285: Status 404 returned error can't find the container with id e1128d370bf75e794b3140153345832b67a0f42a2f50a697c11b417d1532c285
Jul 13 20:40:55.807536 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.807509    2584 generic.go:356] "Generic (PLEG): container finished" podID="898516a6-29a6-4444-a8bd-902c4020c57f" containerID="54af12f16272350f723689889276505b3c0e33a69b07d5ce9c7f38c935df0912" exitCode=7
Jul 13 20:40:55.807812 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.807576    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" event={"ID":"898516a6-29a6-4444-a8bd-902c4020c57f","Type":"ContainerDied","Data":"54af12f16272350f723689889276505b3c0e33a69b07d5ce9c7f38c935df0912"}
Jul 13 20:40:55.807812 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.807611    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" event={"ID":"898516a6-29a6-4444-a8bd-902c4020c57f","Type":"ContainerStarted","Data":"e1128d370bf75e794b3140153345832b67a0f42a2f50a697c11b417d1532c285"}
Jul 13 20:40:55.807812 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:55.807752    2584 scope.go:117] "RemoveContainer" containerID="54af12f16272350f723689889276505b3c0e33a69b07d5ce9c7f38c935df0912"
Jul 13 20:40:56.812606 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:56.812517    2584 generic.go:356] "Generic (PLEG): container finished" podID="898516a6-29a6-4444-a8bd-902c4020c57f" containerID="407695b29c563dbc706ee66822792d941cf0c55b5c5b21331847af3630dd1fbc" exitCode=7
Jul 13 20:40:56.812606 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:56.812593    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" event={"ID":"898516a6-29a6-4444-a8bd-902c4020c57f","Type":"ContainerDied","Data":"407695b29c563dbc706ee66822792d941cf0c55b5c5b21331847af3630dd1fbc"}
Jul 13 20:40:56.813217 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:56.812638    2584 scope.go:117] "RemoveContainer" containerID="54af12f16272350f723689889276505b3c0e33a69b07d5ce9c7f38c935df0912"
Jul 13 20:40:56.813217 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:56.812795    2584 scope.go:117] "RemoveContainer" containerID="407695b29c563dbc706ee66822792d941cf0c55b5c5b21331847af3630dd1fbc"
Jul 13 20:40:56.813217 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:40:56.812982    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v_odh-ai-gateway-infra(898516a6-29a6-4444-a8bd-902c4020c57f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" podUID="898516a6-29a6-4444-a8bd-902c4020c57f"
Jul 13 20:40:57.800200 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:57.800171    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:40:57.817829 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:57.817807    2584 scope.go:117] "RemoveContainer" containerID="407695b29c563dbc706ee66822792d941cf0c55b5c5b21331847af3630dd1fbc"
Jul 13 20:40:57.818248 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:40:57.818002    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v_odh-ai-gateway-infra(898516a6-29a6-4444-a8bd-902c4020c57f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" podUID="898516a6-29a6-4444-a8bd-902c4020c57f"
Jul 13 20:40:58.464916 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:40:58.464878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:41:08.632983 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:08.632947    2584 scope.go:117] "RemoveContainer" containerID="407695b29c563dbc706ee66822792d941cf0c55b5c5b21331847af3630dd1fbc"
Jul 13 20:41:08.861865 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:08.861841    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" event={"ID":"898516a6-29a6-4444-a8bd-902c4020c57f","Type":"ContainerStarted","Data":"a00bd182911f6c84392d658d81b93f92018dc2a63f970b365d03c8566572130b"}
Jul 13 20:41:08.878761 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:08.878717    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" podStartSLOduration=14.878703287 podStartE2EDuration="14.878703287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:41:08.878307843 +0000 UTC m=+1602.779010503" watchObservedRunningTime="2026-07-13 20:41:08.878703287 +0000 UTC m=+1602.779405945"
Jul 13 20:41:09.868015 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:09.867981    2584 generic.go:356] "Generic (PLEG): container finished" podID="898516a6-29a6-4444-a8bd-902c4020c57f" containerID="a00bd182911f6c84392d658d81b93f92018dc2a63f970b365d03c8566572130b" exitCode=0
Jul 13 20:41:09.868388 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:09.868052    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" event={"ID":"898516a6-29a6-4444-a8bd-902c4020c57f","Type":"ContainerDied","Data":"a00bd182911f6c84392d658d81b93f92018dc2a63f970b365d03c8566572130b"}
Jul 13 20:41:09.868388 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:09.868130    2584 scope.go:117] "RemoveContainer" containerID="407695b29c563dbc706ee66822792d941cf0c55b5c5b21331847af3630dd1fbc"
Jul 13 20:41:09.895352 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:09.895331    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"]
Jul 13 20:41:11.010786 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.010761    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:41:11.064049 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.064025    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/898516a6-29a6-4444-a8bd-902c4020c57f-maas-api-service-ca\") pod \"898516a6-29a6-4444-a8bd-902c4020c57f\" (UID: \"898516a6-29a6-4444-a8bd-902c4020c57f\") "
Jul 13 20:41:11.064366 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.064344    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898516a6-29a6-4444-a8bd-902c4020c57f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "898516a6-29a6-4444-a8bd-902c4020c57f" (UID: "898516a6-29a6-4444-a8bd-902c4020c57f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:41:11.165565 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.165501    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/898516a6-29a6-4444-a8bd-902c4020c57f-maas-api-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:41:11.879062 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.879031    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v" event={"ID":"898516a6-29a6-4444-a8bd-902c4020c57f","Type":"ContainerDied","Data":"e1128d370bf75e794b3140153345832b67a0f42a2f50a697c11b417d1532c285"}
Jul 13 20:41:11.879062 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.879059    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"
Jul 13 20:41:11.881183 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.879075    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1128d370bf75e794b3140153345832b67a0f42a2f50a697c11b417d1532c285"
Jul 13 20:41:11.911823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.911800    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"]
Jul 13 20:41:11.915762 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:11.915738    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9ebcdac6-n4j7v"]
Jul 13 20:41:12.637855 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:12.637821    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" path="/var/lib/kubelet/pods/898516a6-29a6-4444-a8bd-902c4020c57f/volumes"
Jul 13 20:41:55.182990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.182958    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"]
Jul 13 20:41:55.183488 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.183198    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" podUID="adeaf95f-0d61-449c-b12f-57940be23835" containerName="maas-api" containerID="cri-o://c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400" gracePeriod=30
Jul 13 20:41:55.211777 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.211745    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:41:55.420114 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.420084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:41:55.575589 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.575527    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jtkr\" (UniqueName: \"kubernetes.io/projected/adeaf95f-0d61-449c-b12f-57940be23835-kube-api-access-5jtkr\") pod \"adeaf95f-0d61-449c-b12f-57940be23835\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") "
Jul 13 20:41:55.575728 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.575612    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adeaf95f-0d61-449c-b12f-57940be23835-openshift-service-ca\") pod \"adeaf95f-0d61-449c-b12f-57940be23835\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") "
Jul 13 20:41:55.575728 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.575666    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls\") pod \"adeaf95f-0d61-449c-b12f-57940be23835\" (UID: \"adeaf95f-0d61-449c-b12f-57940be23835\") "
Jul 13 20:41:55.575908 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.575887    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adeaf95f-0d61-449c-b12f-57940be23835-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adeaf95f-0d61-449c-b12f-57940be23835" (UID: "adeaf95f-0d61-449c-b12f-57940be23835"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:41:55.577679 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.577648    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adeaf95f-0d61-449c-b12f-57940be23835" (UID: "adeaf95f-0d61-449c-b12f-57940be23835"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:41:55.577775 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.577680    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adeaf95f-0d61-449c-b12f-57940be23835-kube-api-access-5jtkr" (OuterVolumeSpecName: "kube-api-access-5jtkr") pod "adeaf95f-0d61-449c-b12f-57940be23835" (UID: "adeaf95f-0d61-449c-b12f-57940be23835"). InnerVolumeSpecName "kube-api-access-5jtkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:41:55.676265 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.676242    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adeaf95f-0d61-449c-b12f-57940be23835-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:41:55.676265 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.676263    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adeaf95f-0d61-449c-b12f-57940be23835-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:41:55.676387 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:55.676273    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtkr\" (UniqueName: \"kubernetes.io/projected/adeaf95f-0d61-449c-b12f-57940be23835-kube-api-access-5jtkr\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:41:56.047358 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.047328    2584 generic.go:356] "Generic (PLEG): container finished" podID="adeaf95f-0d61-449c-b12f-57940be23835" containerID="c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400" exitCode=0
Jul 13 20:41:56.047523 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.047371    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"
Jul 13 20:41:56.047523 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.047369    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" event={"ID":"adeaf95f-0d61-449c-b12f-57940be23835","Type":"ContainerDied","Data":"c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400"}
Jul 13 20:41:56.047523 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.047508    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw" event={"ID":"adeaf95f-0d61-449c-b12f-57940be23835","Type":"ContainerDied","Data":"8743c0a6a6866ea4e75061d67df5117131534c6c381e25264a398ed673bd1c47"}
Jul 13 20:41:56.047623 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.047538    2584 scope.go:117] "RemoveContainer" containerID="c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400"
Jul 13 20:41:56.056647 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.056632    2584 scope.go:117] "RemoveContainer" containerID="c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400"
Jul 13 20:41:56.056893 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:41:56.056873    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400\": container with ID starting with c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400 not found: ID does not exist" containerID="c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400"
Jul 13 20:41:56.056958 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.056902    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400"} err="failed to get container status \"c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400\": rpc error: code = NotFound desc = could not find container \"c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400\": container with ID starting with c697f1d48a1e084dc762c07095e42d9502e8cb38867bde2a9b8820f61de11400 not found: ID does not exist"
Jul 13 20:41:56.068590 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.068558    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"]
Jul 13 20:41:56.073620 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.073600    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ebcdac6-6446c89c7b-mzltw"]
Jul 13 20:41:56.636893 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:56.636864    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adeaf95f-0d61-449c-b12f-57940be23835" path="/var/lib/kubelet/pods/adeaf95f-0d61-449c-b12f-57940be23835/volumes"
Jul 13 20:41:59.839110 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:41:59.839071    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:05.599721 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:05.599687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:06.766831 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:06.766782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:10.917816 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:10.917772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:11.239552 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:11.239522    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:12.455164 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.455120    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:12.549200 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.549167    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:12.616228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616203    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"]
Jul 13 20:42:12.616493 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616479    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616493 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616494    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616585 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616504    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616585 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616509    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616585 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616517    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616585 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616523    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616585 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616535    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adeaf95f-0d61-449c-b12f-57940be23835" containerName="maas-api"
Jul 13 20:42:12.616585 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616541    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="adeaf95f-0d61-449c-b12f-57940be23835" containerName="maas-api"
Jul 13 20:42:12.616765 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616590    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.616765 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616597    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="adeaf95f-0d61-449c-b12f-57940be23835" containerName="maas-api"
Jul 13 20:42:12.616765 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.616606    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:12.620717 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.620700    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.625551 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.625531    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4c03cf2d-serving-cert\""
Jul 13 20:42:12.625658 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.625613    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:42:12.626721 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.626702    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:42:12.626827 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.626758    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:42:12.632324 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.632304    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"]
Jul 13 20:42:12.677051 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.677013    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:12.701176 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.701157    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74bae37-7f07-494f-b1e0-e48555c6dd98-maas-api-tls\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.701276 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.701183    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45lj\" (UniqueName: \"kubernetes.io/projected/c74bae37-7f07-494f-b1e0-e48555c6dd98-kube-api-access-k45lj\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.701276 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.701225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bae37-7f07-494f-b1e0-e48555c6dd98-openshift-service-ca\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.802195 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.802141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bae37-7f07-494f-b1e0-e48555c6dd98-openshift-service-ca\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.802286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.802200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74bae37-7f07-494f-b1e0-e48555c6dd98-maas-api-tls\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.802286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.802220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45lj\" (UniqueName: \"kubernetes.io/projected/c74bae37-7f07-494f-b1e0-e48555c6dd98-kube-api-access-k45lj\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.802844 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.802811    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bae37-7f07-494f-b1e0-e48555c6dd98-openshift-service-ca\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.804800 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.804782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74bae37-7f07-494f-b1e0-e48555c6dd98-maas-api-tls\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.812499 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.812473    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45lj\" (UniqueName: \"kubernetes.io/projected/c74bae37-7f07-494f-b1e0-e48555c6dd98-kube-api-access-k45lj\") pod \"maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:12.828956 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.828935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:12.930674 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:12.930651    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:13.098977 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:42:13.098947    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74bae37_7f07_494f_b1e0_e48555c6dd98.slice/crio-0e87968c7d3905d1fc4f444722b47155059a842d504beab96eba7267b615351a WatchSource:0}: Error finding container 0e87968c7d3905d1fc4f444722b47155059a842d504beab96eba7267b615351a: Status 404 returned error can't find the container with id 0e87968c7d3905d1fc4f444722b47155059a842d504beab96eba7267b615351a
Jul 13 20:42:13.110080 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:13.110057    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"]
Jul 13 20:42:13.113529 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:13.113505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc" event={"ID":"c74bae37-7f07-494f-b1e0-e48555c6dd98","Type":"ContainerStarted","Data":"0e87968c7d3905d1fc4f444722b47155059a842d504beab96eba7267b615351a"}
Jul 13 20:42:13.348148 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:13.348076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:13.439888 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:13.439849    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:15.122980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:15.122944    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc" event={"ID":"c74bae37-7f07-494f-b1e0-e48555c6dd98","Type":"ContainerStarted","Data":"2628fad02005d6af948ec95ecbbda14bffc5bd2ca94002a6c68c02a62ae37344"}
Jul 13 20:42:15.123451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:15.123103    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:15.146108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:15.145948    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc" podStartSLOduration=2.0512424400000002 podStartE2EDuration="3.145933486s" totalImagesPullingTime="1.094691046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:42:15.145088484 +0000 UTC m=+1669.045791147" watchObservedRunningTime="2026-07-13 20:42:15.145933486 +0000 UTC m=+1669.046636150"
Jul 13 20:42:16.757592 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.757557    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"]
Jul 13 20:42:16.757933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.757878    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="898516a6-29a6-4444-a8bd-902c4020c57f" containerName="revoke-keys"
Jul 13 20:42:16.760821 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.760806    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:16.764093 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.764070    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:42:16.768448 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.768424    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"]
Jul 13 20:42:16.824539 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.824516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb89182d-a71d-4010-801a-a6faf5291cc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn\" (UID: \"eb89182d-a71d-4010-801a-a6faf5291cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:16.925763 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.925739    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb89182d-a71d-4010-801a-a6faf5291cc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn\" (UID: \"eb89182d-a71d-4010-801a-a6faf5291cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:16.926307 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:16.926284    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb89182d-a71d-4010-801a-a6faf5291cc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn\" (UID: \"eb89182d-a71d-4010-801a-a6faf5291cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:17.071750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:17.071689    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:17.199052 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:17.199024    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"]
Jul 13 20:42:17.201576 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:42:17.201554    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb89182d_a71d_4010_801a_a6faf5291cc3.slice/crio-58b9a393d39497121e539084e6be4238d687fdfe203062029d567369bc84398a WatchSource:0}: Error finding container 58b9a393d39497121e539084e6be4238d687fdfe203062029d567369bc84398a: Status 404 returned error can't find the container with id 58b9a393d39497121e539084e6be4238d687fdfe203062029d567369bc84398a
Jul 13 20:42:17.890370 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:17.890344    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:17.962941 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:17.962909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:18.138010 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:18.137963    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerID="f3795643aa7144d2a6c501171a24ef591a6424074921520e42a978ddeca854a3" exitCode=7
Jul 13 20:42:18.138198 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:18.138055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" event={"ID":"eb89182d-a71d-4010-801a-a6faf5291cc3","Type":"ContainerDied","Data":"f3795643aa7144d2a6c501171a24ef591a6424074921520e42a978ddeca854a3"}
Jul 13 20:42:18.138198 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:18.138107    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" event={"ID":"eb89182d-a71d-4010-801a-a6faf5291cc3","Type":"ContainerStarted","Data":"58b9a393d39497121e539084e6be4238d687fdfe203062029d567369bc84398a"}
Jul 13 20:42:18.138320 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:18.138231    2584 scope.go:117] "RemoveContainer" containerID="f3795643aa7144d2a6c501171a24ef591a6424074921520e42a978ddeca854a3"
Jul 13 20:42:19.143498 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:19.143463    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerID="16b542ee6c6ce57febd233e8f0b24d6ffe9c724a28b395c4adfd1d68b573a440" exitCode=7
Jul 13 20:42:19.143948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:19.143518    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" event={"ID":"eb89182d-a71d-4010-801a-a6faf5291cc3","Type":"ContainerDied","Data":"16b542ee6c6ce57febd233e8f0b24d6ffe9c724a28b395c4adfd1d68b573a440"}
Jul 13 20:42:19.143948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:19.143585    2584 scope.go:117] "RemoveContainer" containerID="f3795643aa7144d2a6c501171a24ef591a6424074921520e42a978ddeca854a3"
Jul 13 20:42:19.143948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:19.143805    2584 scope.go:117] "RemoveContainer" containerID="16b542ee6c6ce57febd233e8f0b24d6ffe9c724a28b395c4adfd1d68b573a440"
Jul 13 20:42:19.144064 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:42:19.144026    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn_odh-ai-gateway-infra(eb89182d-a71d-4010-801a-a6faf5291cc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3"
Jul 13 20:42:20.149511 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:20.149488    2584 scope.go:117] "RemoveContainer" containerID="16b542ee6c6ce57febd233e8f0b24d6ffe9c724a28b395c4adfd1d68b573a440"
Jul 13 20:42:20.149849 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:42:20.149657    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn_odh-ai-gateway-infra(eb89182d-a71d-4010-801a-a6faf5291cc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3"
Jul 13 20:42:21.131952 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:21.131921    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:42:21.233515 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:21.233483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:35.632144 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:35.632105    2584 scope.go:117] "RemoveContainer" containerID="16b542ee6c6ce57febd233e8f0b24d6ffe9c724a28b395c4adfd1d68b573a440"
Jul 13 20:42:36.212911 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:36.212880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" event={"ID":"eb89182d-a71d-4010-801a-a6faf5291cc3","Type":"ContainerStarted","Data":"da5797f1d226fd01540702b02d7e7171d9c6da5ed539f095677e3e4b1064fd4e"}
Jul 13 20:42:36.230370 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:36.230326    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" podStartSLOduration=20.230312367 podStartE2EDuration="20.230312367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:42:36.22866043 +0000 UTC m=+1690.129363093" watchObservedRunningTime="2026-07-13 20:42:36.230312367 +0000 UTC m=+1690.131015029"
Jul 13 20:42:37.218920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:37.218878    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerID="da5797f1d226fd01540702b02d7e7171d9c6da5ed539f095677e3e4b1064fd4e" exitCode=0
Jul 13 20:42:37.219347 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:37.218951    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" event={"ID":"eb89182d-a71d-4010-801a-a6faf5291cc3","Type":"ContainerDied","Data":"da5797f1d226fd01540702b02d7e7171d9c6da5ed539f095677e3e4b1064fd4e"}
Jul 13 20:42:37.219347 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:37.219023    2584 scope.go:117] "RemoveContainer" containerID="16b542ee6c6ce57febd233e8f0b24d6ffe9c724a28b395c4adfd1d68b573a440"
Jul 13 20:42:37.245933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:37.245910    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"]
Jul 13 20:42:38.354934 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:38.354905    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:38.384707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:38.384683    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb89182d-a71d-4010-801a-a6faf5291cc3-maas-api-service-ca\") pod \"eb89182d-a71d-4010-801a-a6faf5291cc3\" (UID: \"eb89182d-a71d-4010-801a-a6faf5291cc3\") "
Jul 13 20:42:38.385032 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:38.385006    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb89182d-a71d-4010-801a-a6faf5291cc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb89182d-a71d-4010-801a-a6faf5291cc3" (UID: "eb89182d-a71d-4010-801a-a6faf5291cc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:42:38.486266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:38.486209    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb89182d-a71d-4010-801a-a6faf5291cc3-maas-api-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:42:39.230993 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:39.230967    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"
Jul 13 20:42:39.231165 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:39.230964    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn" event={"ID":"eb89182d-a71d-4010-801a-a6faf5291cc3","Type":"ContainerDied","Data":"58b9a393d39497121e539084e6be4238d687fdfe203062029d567369bc84398a"}
Jul 13 20:42:39.231165 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:39.231118    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b9a393d39497121e539084e6be4238d687fdfe203062029d567369bc84398a"
Jul 13 20:42:39.262486 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:39.262457    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"]
Jul 13 20:42:39.266101 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:39.266075    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-p8lfn"]
Jul 13 20:42:40.636961 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:42:40.636922    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" path="/var/lib/kubelet/pods/eb89182d-a71d-4010-801a-a6faf5291cc3/volumes"
Jul 13 20:43:16.802017 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.801934    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"]
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802300    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802318    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802332    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802341    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802350    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802358    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802467    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802481    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.802588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.802489    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb89182d-a71d-4010-801a-a6faf5291cc3" containerName="revoke-keys"
Jul 13 20:43:16.805528 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.805508    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:16.807978 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.807960    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:43:16.811360 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.811338    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"]
Jul 13 20:43:16.958379 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:16.958350    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490644b6-1023-4a8b-8ad2-3c3e5d415eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n\" (UID: \"490644b6-1023-4a8b-8ad2-3c3e5d415eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:17.058991 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:17.058935    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490644b6-1023-4a8b-8ad2-3c3e5d415eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n\" (UID: \"490644b6-1023-4a8b-8ad2-3c3e5d415eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:17.059491 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:17.059472    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490644b6-1023-4a8b-8ad2-3c3e5d415eb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n\" (UID: \"490644b6-1023-4a8b-8ad2-3c3e5d415eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:17.116557 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:17.116536    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:17.250495 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:17.250421    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"]
Jul 13 20:43:17.252643 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:43:17.252617    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490644b6_1023_4a8b_8ad2_3c3e5d415eb3.slice/crio-f233059f99b71b77bfc5e6b21a4f7f0bd91938248e49164ea34e59f23b50d13e WatchSource:0}: Error finding container f233059f99b71b77bfc5e6b21a4f7f0bd91938248e49164ea34e59f23b50d13e: Status 404 returned error can't find the container with id f233059f99b71b77bfc5e6b21a4f7f0bd91938248e49164ea34e59f23b50d13e
Jul 13 20:43:17.380581 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:17.380545    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n" event={"ID":"490644b6-1023-4a8b-8ad2-3c3e5d415eb3","Type":"ContainerStarted","Data":"f233059f99b71b77bfc5e6b21a4f7f0bd91938248e49164ea34e59f23b50d13e"}
Jul 13 20:43:18.386455 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:18.386416    2584 generic.go:356] "Generic (PLEG): container finished" podID="490644b6-1023-4a8b-8ad2-3c3e5d415eb3" containerID="bac9fbeb0c49b5f22b9b11a01f4a69f8b3ee4e22de8b1e9d86b24e8fd6a93b09" exitCode=0
Jul 13 20:43:18.386825 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:18.386488    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n" event={"ID":"490644b6-1023-4a8b-8ad2-3c3e5d415eb3","Type":"ContainerDied","Data":"bac9fbeb0c49b5f22b9b11a01f4a69f8b3ee4e22de8b1e9d86b24e8fd6a93b09"}
Jul 13 20:43:19.512534 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:19.512504    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:19.575100 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:19.575080    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490644b6-1023-4a8b-8ad2-3c3e5d415eb3-maas-api-service-ca\") pod \"490644b6-1023-4a8b-8ad2-3c3e5d415eb3\" (UID: \"490644b6-1023-4a8b-8ad2-3c3e5d415eb3\") "
Jul 13 20:43:19.575468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:19.575437    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490644b6-1023-4a8b-8ad2-3c3e5d415eb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "490644b6-1023-4a8b-8ad2-3c3e5d415eb3" (UID: "490644b6-1023-4a8b-8ad2-3c3e5d415eb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:43:19.676136 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:19.676081    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/490644b6-1023-4a8b-8ad2-3c3e5d415eb3-maas-api-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:43:20.395862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:20.395825    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n" event={"ID":"490644b6-1023-4a8b-8ad2-3c3e5d415eb3","Type":"ContainerDied","Data":"f233059f99b71b77bfc5e6b21a4f7f0bd91938248e49164ea34e59f23b50d13e"}
Jul 13 20:43:20.395862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:20.395856    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n"
Jul 13 20:43:20.396062 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:20.395875    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f233059f99b71b77bfc5e6b21a4f7f0bd91938248e49164ea34e59f23b50d13e"
Jul 13 20:43:27.202923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.202887    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"]
Jul 13 20:43:27.203368 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.203126    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc" podUID="c74bae37-7f07-494f-b1e0-e48555c6dd98" containerName="maas-api" containerID="cri-o://2628fad02005d6af948ec95ecbbda14bffc5bd2ca94002a6c68c02a62ae37344" gracePeriod=30
Jul 13 20:43:27.226520 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.226483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:27.425070 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.425045    2584 generic.go:356] "Generic (PLEG): container finished" podID="c74bae37-7f07-494f-b1e0-e48555c6dd98" containerID="2628fad02005d6af948ec95ecbbda14bffc5bd2ca94002a6c68c02a62ae37344" exitCode=0
Jul 13 20:43:27.425181 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.425119    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc" event={"ID":"c74bae37-7f07-494f-b1e0-e48555c6dd98","Type":"ContainerDied","Data":"2628fad02005d6af948ec95ecbbda14bffc5bd2ca94002a6c68c02a62ae37344"}
Jul 13 20:43:27.425181 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.425163    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc" event={"ID":"c74bae37-7f07-494f-b1e0-e48555c6dd98","Type":"ContainerDied","Data":"0e87968c7d3905d1fc4f444722b47155059a842d504beab96eba7267b615351a"}
Jul 13 20:43:27.425181 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.425181    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e87968c7d3905d1fc4f444722b47155059a842d504beab96eba7267b615351a"
Jul 13 20:43:27.435599 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.435579    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:43:27.528098 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.528045    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74bae37-7f07-494f-b1e0-e48555c6dd98-maas-api-tls\") pod \"c74bae37-7f07-494f-b1e0-e48555c6dd98\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") "
Jul 13 20:43:27.528098 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.528076    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k45lj\" (UniqueName: \"kubernetes.io/projected/c74bae37-7f07-494f-b1e0-e48555c6dd98-kube-api-access-k45lj\") pod \"c74bae37-7f07-494f-b1e0-e48555c6dd98\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") "
Jul 13 20:43:27.528266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.528110    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bae37-7f07-494f-b1e0-e48555c6dd98-openshift-service-ca\") pod \"c74bae37-7f07-494f-b1e0-e48555c6dd98\" (UID: \"c74bae37-7f07-494f-b1e0-e48555c6dd98\") "
Jul 13 20:43:27.528534 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.528511    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74bae37-7f07-494f-b1e0-e48555c6dd98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c74bae37-7f07-494f-b1e0-e48555c6dd98" (UID: "c74bae37-7f07-494f-b1e0-e48555c6dd98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:43:27.530236 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.530206    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74bae37-7f07-494f-b1e0-e48555c6dd98-kube-api-access-k45lj" (OuterVolumeSpecName: "kube-api-access-k45lj") pod "c74bae37-7f07-494f-b1e0-e48555c6dd98" (UID: "c74bae37-7f07-494f-b1e0-e48555c6dd98"). InnerVolumeSpecName "kube-api-access-k45lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:43:27.530322 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.530239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74bae37-7f07-494f-b1e0-e48555c6dd98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c74bae37-7f07-494f-b1e0-e48555c6dd98" (UID: "c74bae37-7f07-494f-b1e0-e48555c6dd98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:43:27.628618 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.628595    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c74bae37-7f07-494f-b1e0-e48555c6dd98-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:43:27.628704 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.628618    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k45lj\" (UniqueName: \"kubernetes.io/projected/c74bae37-7f07-494f-b1e0-e48555c6dd98-kube-api-access-k45lj\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:43:27.628704 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:27.628633    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c74bae37-7f07-494f-b1e0-e48555c6dd98-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:43:28.429969 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:28.429936    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"
Jul 13 20:43:28.453088 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:28.453060    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"]
Jul 13 20:43:28.456655 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:28.456630    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4c03cf2d-58cfb494f9-z9czc"]
Jul 13 20:43:28.637403 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:28.637365    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74bae37-7f07-494f-b1e0-e48555c6dd98" path="/var/lib/kubelet/pods/c74bae37-7f07-494f-b1e0-e48555c6dd98/volumes"
Jul 13 20:43:31.879581 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:31.879541    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:33.710263 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:33.710229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:33.811950 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:33.811922    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:37.494110 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:37.494076    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:37.570222 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:37.570196    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:42.079804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:42.079763    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:42.677081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:42.677048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:42.773731 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:42.773696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:42.876948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:42.876918    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:44.344194 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:44.344164    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:48.622974 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:48.622942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:50.096899 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:50.096869    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:50.196954 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:50.196920    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:50.298849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:50.298820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:50.397213 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:50.397139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:50.852253 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:43:50.852217    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:16.795526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:16.795432    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:22.742565 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:22.742529    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:22.850822 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:22.850793    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:22.943708 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:22.943671    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:38.830122 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:38.830084    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:44.659034 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:44.658987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:44.760405 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:44.760371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:44.858645 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:44.858617    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:55.363745 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:55.363712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:56.188301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:44:56.188265    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:00.137665 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137632    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"]
Jul 13 20:45:00.138081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137887    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="490644b6-1023-4a8b-8ad2-3c3e5d415eb3" containerName="revoke-keys"
Jul 13 20:45:00.138081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137897    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="490644b6-1023-4a8b-8ad2-3c3e5d415eb3" containerName="revoke-keys"
Jul 13 20:45:00.138081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137917    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74bae37-7f07-494f-b1e0-e48555c6dd98" containerName="maas-api"
Jul 13 20:45:00.138081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137924    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74bae37-7f07-494f-b1e0-e48555c6dd98" containerName="maas-api"
Jul 13 20:45:00.138081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137969    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74bae37-7f07-494f-b1e0-e48555c6dd98" containerName="maas-api"
Jul 13 20:45:00.138081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.137977    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="490644b6-1023-4a8b-8ad2-3c3e5d415eb3" containerName="revoke-keys"
Jul 13 20:45:00.140903 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.140886    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"
Jul 13 20:45:00.144224 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.144202    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:45:00.149476 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.149451    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"]
Jul 13 20:45:00.151351 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.151333    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"
Jul 13 20:45:00.280182 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.280148    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"]
Jul 13 20:45:00.283333 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:45:00.283304    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdea227_b6c5_4f66_8004_8d1746aeabc8.slice/crio-d2c7edcb263e5c19093923ba5543c6db9326d3b24cae843bff49797419436b58 WatchSource:0}: Error finding container d2c7edcb263e5c19093923ba5543c6db9326d3b24cae843bff49797419436b58: Status 404 returned error can't find the container with id d2c7edcb263e5c19093923ba5543c6db9326d3b24cae843bff49797419436b58
Jul 13 20:45:00.800505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.800425    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv" event={"ID":"0cdea227-b6c5-4f66-8004-8d1746aeabc8","Type":"ContainerStarted","Data":"d4cecf5ac97d0f02d0fb315ebb405c6b657908e29cc2d13ee6f7cc319660d208"}
Jul 13 20:45:00.800505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.800477    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv" event={"ID":"0cdea227-b6c5-4f66-8004-8d1746aeabc8","Type":"ContainerStarted","Data":"d2c7edcb263e5c19093923ba5543c6db9326d3b24cae843bff49797419436b58"}
Jul 13 20:45:00.816774 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:00.816718    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv" podStartSLOduration=0.816703728 podStartE2EDuration="816.703728ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:45:00.815603519 +0000 UTC m=+1834.716306183" watchObservedRunningTime="2026-07-13 20:45:00.816703728 +0000 UTC m=+1834.717406390"
Jul 13 20:45:01.806069 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:01.806034    2584 generic.go:356] "Generic (PLEG): container finished" podID="0cdea227-b6c5-4f66-8004-8d1746aeabc8" containerID="d4cecf5ac97d0f02d0fb315ebb405c6b657908e29cc2d13ee6f7cc319660d208" exitCode=0
Jul 13 20:45:01.806488 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:01.806097    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv" event={"ID":"0cdea227-b6c5-4f66-8004-8d1746aeabc8","Type":"ContainerDied","Data":"d4cecf5ac97d0f02d0fb315ebb405c6b657908e29cc2d13ee6f7cc319660d208"}
Jul 13 20:45:01.884326 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:01.884289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:01.985719 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:01.985687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:02.083377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:02.083290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:02.269789 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:02.269753    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:02.935562 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:02.935512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:02.936581 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:02.936556    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"
Jul 13 20:45:03.123697 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:03.123666    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:03.224230 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:03.224200    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:03.815528 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:03.815491    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv" event={"ID":"0cdea227-b6c5-4f66-8004-8d1746aeabc8","Type":"ContainerDied","Data":"d2c7edcb263e5c19093923ba5543c6db9326d3b24cae843bff49797419436b58"}
Jul 13 20:45:03.815528 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:03.815517    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732925-rr9cv"
Jul 13 20:45:03.815716 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:03.815548    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c7edcb263e5c19093923ba5543c6db9326d3b24cae843bff49797419436b58"
Jul 13 20:45:04.906209 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:04.906176    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:10.465805 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:10.465767    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:10.562702 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:10.562662    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:11.186338 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:11.186307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:16.915731 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:16.915697    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:17.018980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:17.018943    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:17.215987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:17.215908    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:17.498227 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:17.498138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:17.739829 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:17.739792    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:20.361583 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.361543    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:20.428692 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.428664    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"]
Jul 13 20:45:20.429054 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.429040    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cdea227-b6c5-4f66-8004-8d1746aeabc8" containerName="cleanup"
Jul 13 20:45:20.429096 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.429059    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cdea227-b6c5-4f66-8004-8d1746aeabc8" containerName="cleanup"
Jul 13 20:45:20.429160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.429149    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cdea227-b6c5-4f66-8004-8d1746aeabc8" containerName="cleanup"
Jul 13 20:45:20.432216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.432195    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.435135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.435109    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:45:20.435244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.435118    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-19b1db0a-serving-cert\""
Jul 13 20:45:20.435244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.435123    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:45:20.436286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.436265    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:45:20.441379 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.441356    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"]
Jul 13 20:45:20.453099 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.453078    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.453195 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.453113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c306e3-a83e-4d42-9d07-5818b0363613-openshift-service-ca\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.453195 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.453158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6qm\" (UniqueName: \"kubernetes.io/projected/46c306e3-a83e-4d42-9d07-5818b0363613-kube-api-access-ws6qm\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.454160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.454138    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:20.553856 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.553833    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6qm\" (UniqueName: \"kubernetes.io/projected/46c306e3-a83e-4d42-9d07-5818b0363613-kube-api-access-ws6qm\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.553963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.553894    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.553963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.553927    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c306e3-a83e-4d42-9d07-5818b0363613-openshift-service-ca\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.554076 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:45:20.554013    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-serving-cert: secret "maas-api-e2e-mt-19b1db0a-serving-cert" not found
Jul 13 20:45:20.554129 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:45:20.554093    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls podName:46c306e3-a83e-4d42-9d07-5818b0363613 nodeName:}" failed. No retries permitted until 2026-07-13 20:45:21.054073684 +0000 UTC m=+1854.954776324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls") pod "maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" (UID: "46c306e3-a83e-4d42-9d07-5818b0363613") : secret "maas-api-e2e-mt-19b1db0a-serving-cert" not found
Jul 13 20:45:20.554569 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.554548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c306e3-a83e-4d42-9d07-5818b0363613-openshift-service-ca\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:20.554781 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.554757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:20.562941 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:20.562915    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6qm\" (UniqueName: \"kubernetes.io/projected/46c306e3-a83e-4d42-9d07-5818b0363613-kube-api-access-ws6qm\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:21.042505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.042470    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:21.056743 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.056718    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:21.059262 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.059236    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls\") pod \"maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:21.139780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.139750    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:21.343273 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.343247    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:21.471826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.471803    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"]
Jul 13 20:45:21.473852 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:45:21.473825    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c306e3_a83e_4d42_9d07_5818b0363613.slice/crio-dfec992453c2c36190d06609234d38187f186fd49048d0cd702820cd30e1db8a WatchSource:0}: Error finding container dfec992453c2c36190d06609234d38187f186fd49048d0cd702820cd30e1db8a: Status 404 returned error can't find the container with id dfec992453c2c36190d06609234d38187f186fd49048d0cd702820cd30e1db8a
Jul 13 20:45:21.475139 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.475121    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:45:21.893766 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:21.893729    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" event={"ID":"46c306e3-a83e-4d42-9d07-5818b0363613","Type":"ContainerStarted","Data":"dfec992453c2c36190d06609234d38187f186fd49048d0cd702820cd30e1db8a"}
Jul 13 20:45:22.899322 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:22.899288    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" event={"ID":"46c306e3-a83e-4d42-9d07-5818b0363613","Type":"ContainerStarted","Data":"4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b"}
Jul 13 20:45:22.899707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:22.899431    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:22.916073 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:22.916024    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" podStartSLOduration=2.451966157 podStartE2EDuration="2.916010763s" totalImagesPullingTime="464.044606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:45:22.914503015 +0000 UTC m=+1856.815205676" watchObservedRunningTime="2026-07-13 20:45:22.916010763 +0000 UTC m=+1856.816713424"
Jul 13 20:45:24.862803 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:24.862773    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:25.532811 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:25.532772    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:25.600036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:25.600008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:25.951352 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:25.951319    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:28.909291 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:28.909253    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:31.693947 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:31.693913    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:31.794743 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:31.794708    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:42.406838 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.406798    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"]
Jul 13 20:45:42.407286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.407050    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" podUID="46c306e3-a83e-4d42-9d07-5818b0363613" containerName="maas-api" containerID="cri-o://4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b" gracePeriod=30
Jul 13 20:45:42.442564 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.442539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:42.647058 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.647036    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:42.717219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.717158    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c306e3-a83e-4d42-9d07-5818b0363613-openshift-service-ca\") pod \"46c306e3-a83e-4d42-9d07-5818b0363613\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") "
Jul 13 20:45:42.717219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.717201    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls\") pod \"46c306e3-a83e-4d42-9d07-5818b0363613\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") "
Jul 13 20:45:42.717219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.717219    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws6qm\" (UniqueName: \"kubernetes.io/projected/46c306e3-a83e-4d42-9d07-5818b0363613-kube-api-access-ws6qm\") pod \"46c306e3-a83e-4d42-9d07-5818b0363613\" (UID: \"46c306e3-a83e-4d42-9d07-5818b0363613\") "
Jul 13 20:45:42.717567 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.717543    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c306e3-a83e-4d42-9d07-5818b0363613-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46c306e3-a83e-4d42-9d07-5818b0363613" (UID: "46c306e3-a83e-4d42-9d07-5818b0363613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:45:42.719355 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.719333    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c306e3-a83e-4d42-9d07-5818b0363613-kube-api-access-ws6qm" (OuterVolumeSpecName: "kube-api-access-ws6qm") pod "46c306e3-a83e-4d42-9d07-5818b0363613" (UID: "46c306e3-a83e-4d42-9d07-5818b0363613"). InnerVolumeSpecName "kube-api-access-ws6qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:45:42.719425 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.719381    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46c306e3-a83e-4d42-9d07-5818b0363613" (UID: "46c306e3-a83e-4d42-9d07-5818b0363613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:45:42.818695 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.818667    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c306e3-a83e-4d42-9d07-5818b0363613-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:45:42.818695 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.818690    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c306e3-a83e-4d42-9d07-5818b0363613-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:45:42.818695 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.818699    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6qm\" (UniqueName: \"kubernetes.io/projected/46c306e3-a83e-4d42-9d07-5818b0363613-kube-api-access-ws6qm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:45:42.985980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.985895    2584 generic.go:356] "Generic (PLEG): container finished" podID="46c306e3-a83e-4d42-9d07-5818b0363613" containerID="4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b" exitCode=0
Jul 13 20:45:42.985980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.985944    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"
Jul 13 20:45:42.985980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.985963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" event={"ID":"46c306e3-a83e-4d42-9d07-5818b0363613","Type":"ContainerDied","Data":"4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b"}
Jul 13 20:45:42.986227 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.986019    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph" event={"ID":"46c306e3-a83e-4d42-9d07-5818b0363613","Type":"ContainerDied","Data":"dfec992453c2c36190d06609234d38187f186fd49048d0cd702820cd30e1db8a"}
Jul 13 20:45:42.986227 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.986047    2584 scope.go:117] "RemoveContainer" containerID="4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b"
Jul 13 20:45:42.994674 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.994657    2584 scope.go:117] "RemoveContainer" containerID="4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b"
Jul 13 20:45:42.994891 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:45:42.994873    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b\": container with ID starting with 4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b not found: ID does not exist" containerID="4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b"
Jul 13 20:45:42.994933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:42.994898    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b"} err="failed to get container status \"4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b\": rpc error: code = NotFound desc = could not find container \"4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b\": container with ID starting with 4968e5a83536f311c0fa52a103b3ede17464b2933d1fb1fd7722dc998b1a282b not found: ID does not exist"
Jul 13 20:45:43.007044 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:43.007014    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"]
Jul 13 20:45:43.010618 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:43.010597    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-19b1db0a-6d955dd548-pqzph"]
Jul 13 20:45:44.638641 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:44.638605    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46c306e3-a83e-4d42-9d07-5818b0363613" path="/var/lib/kubelet/pods/46c306e3-a83e-4d42-9d07-5818b0363613/volumes"
Jul 13 20:45:47.115782 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:47.115699    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:52.930913 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:52.930871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:52.993658 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:52.993629    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:57.834976 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:57.834936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:58.057682 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:58.057643    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:58.180131 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:58.180096    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:58.280208 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:58.280171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:45:58.476017 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:45:58.475949    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:04.100974 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:04.100939    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:04.204810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:04.204781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:04.300026 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:04.299998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:04.445451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:04.445365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:10.245250 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:10.245215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:10.307715 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:10.307690    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:17.085084 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:17.085050    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:17.393864 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:17.393831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:17.908622 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:17.908579    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:20.715294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.715246    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:20.786857 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.786830    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"]
Jul 13 20:46:20.787108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.787096    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c306e3-a83e-4d42-9d07-5818b0363613" containerName="maas-api"
Jul 13 20:46:20.787150 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.787113    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c306e3-a83e-4d42-9d07-5818b0363613" containerName="maas-api"
Jul 13 20:46:20.787186 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.787175    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="46c306e3-a83e-4d42-9d07-5818b0363613" containerName="maas-api"
Jul 13 20:46:20.789966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.789946    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.792626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.792598    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d387c72d-serving-cert\""
Jul 13 20:46:20.792912 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.792885    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:46:20.793015 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.792908    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:46:20.793635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.793612    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:20.793953 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.793936    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:46:20.802143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.802120    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"]
Jul 13 20:46:20.860267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.860239    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:20.892905 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.892881    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkzwn\" (UniqueName: \"kubernetes.io/projected/00854103-6356-4595-aecd-5e137d1f1ca1-kube-api-access-kkzwn\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.892990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.892928    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00854103-6356-4595-aecd-5e137d1f1ca1-maas-api-tls\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.892990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.892951    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00854103-6356-4595-aecd-5e137d1f1ca1-openshift-service-ca\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.962119 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.962093    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:20.994082 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.994031    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00854103-6356-4595-aecd-5e137d1f1ca1-maas-api-tls\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.994082 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.994056    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00854103-6356-4595-aecd-5e137d1f1ca1-openshift-service-ca\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.994219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.994096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkzwn\" (UniqueName: \"kubernetes.io/projected/00854103-6356-4595-aecd-5e137d1f1ca1-kube-api-access-kkzwn\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.994653 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.994634    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00854103-6356-4595-aecd-5e137d1f1ca1-openshift-service-ca\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:20.996552 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:20.996533    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00854103-6356-4595-aecd-5e137d1f1ca1-maas-api-tls\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:21.002259 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:21.002235    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkzwn\" (UniqueName: \"kubernetes.io/projected/00854103-6356-4595-aecd-5e137d1f1ca1-kube-api-access-kkzwn\") pod \"maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:21.100770 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:21.100747    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:21.237958 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:21.237937    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"]
Jul 13 20:46:21.240725 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:46:21.240699    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00854103_6356_4595_aecd_5e137d1f1ca1.slice/crio-6b60fcb97c700fdfea731e5704d015fadef6384a0af8ba59abc95bbd6f9bdcc1 WatchSource:0}: Error finding container 6b60fcb97c700fdfea731e5704d015fadef6384a0af8ba59abc95bbd6f9bdcc1: Status 404 returned error can't find the container with id 6b60fcb97c700fdfea731e5704d015fadef6384a0af8ba59abc95bbd6f9bdcc1
Jul 13 20:46:21.359121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:21.359094    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:21.460626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:21.460596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:22.157267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:22.157238    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr" event={"ID":"00854103-6356-4595-aecd-5e137d1f1ca1","Type":"ContainerStarted","Data":"34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4"}
Jul 13 20:46:22.157642 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:22.157287    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr" event={"ID":"00854103-6356-4595-aecd-5e137d1f1ca1","Type":"ContainerStarted","Data":"6b60fcb97c700fdfea731e5704d015fadef6384a0af8ba59abc95bbd6f9bdcc1"}
Jul 13 20:46:22.157642 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:22.157323    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:22.174364 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:22.174321    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr" podStartSLOduration=1.742533876 podStartE2EDuration="2.174307569s" totalImagesPullingTime="431.773693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:46:22.173055514 +0000 UTC m=+1916.073758175" watchObservedRunningTime="2026-07-13 20:46:22.174307569 +0000 UTC m=+1916.075010231"
Jul 13 20:46:25.215551 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:25.215510    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:28.166056 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:28.166029    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:46:29.594884 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.594852    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"]
Jul 13 20:46:29.598149 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.598126    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.601894 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.601872    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d387c72d\"/\"e2e-lifecycle-model-d387c72d-kserve-self-signed-certs\""
Jul 13 20:46:29.602015 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.601926    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d387c72d\"/\"default-dockercfg-2tlcx\""
Jul 13 20:46:29.602093 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.602009    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d387c72d\"/\"kube-root-ca.crt\""
Jul 13 20:46:29.602093 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.602012    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d387c72d\"/\"openshift-service-ca.crt\""
Jul 13 20:46:29.613209 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.613188    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"]
Jul 13 20:46:29.614524 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.614495    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:46:29.757538 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.757512    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tmp-dir\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.757635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.757550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nkn\" (UniqueName: \"kubernetes.io/projected/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-kube-api-access-98nkn\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.757635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.757577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-dshm\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.757635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.757600    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-model-cache\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.757635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.757622    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tls-certs\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.757794 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.757644    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-home\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.858808 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.858750    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-dshm\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.858808 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.858778    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-model-cache\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.858808 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.858796    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tls-certs\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.859000 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.858818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-home\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.859000 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.858861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tmp-dir\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.859000 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.858884    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nkn\" (UniqueName: \"kubernetes.io/projected/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-kube-api-access-98nkn\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.859276 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.859252    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-model-cache\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.859343 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.859272    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tmp-dir\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.859343 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.859300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-home\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.861039 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.861021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-dshm\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.861306 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.861287    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tls-certs\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.867317 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.867298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nkn\" (UniqueName: \"kubernetes.io/projected/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-kube-api-access-98nkn\") pod \"e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") " pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:29.907693 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:29.907670    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:30.032885 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:30.032855    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"]
Jul 13 20:46:30.036073 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:46:30.036041    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84d26ef5_adc2_4a84_8a2f_9dd0e6140cab.slice/crio-75f1d09c6ec39a18fe8eea52763f2780c9810845ac7c3fa5fef63e1c04cad150 WatchSource:0}: Error finding container 75f1d09c6ec39a18fe8eea52763f2780c9810845ac7c3fa5fef63e1c04cad150: Status 404 returned error can't find the container with id 75f1d09c6ec39a18fe8eea52763f2780c9810845ac7c3fa5fef63e1c04cad150
Jul 13 20:46:30.191143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:30.191066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw" event={"ID":"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab","Type":"ContainerStarted","Data":"0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06"}
Jul 13 20:46:30.191143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:30.191103    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw" event={"ID":"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab","Type":"ContainerStarted","Data":"75f1d09c6ec39a18fe8eea52763f2780c9810845ac7c3fa5fef63e1c04cad150"}
Jul 13 20:46:30.207184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:30.207146    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw" podStartSLOduration=1.2071314690000001 podStartE2EDuration="1.207131469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:46:30.205755999 +0000 UTC m=+1924.106458677" watchObservedRunningTime="2026-07-13 20:46:30.207131469 +0000 UTC m=+1924.107834131"
Jul 13 20:46:39.907926 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:39.907892    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:39.908349 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:39.907937    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:39.920135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:39.920112    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:40.242535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:40.242461    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:46:58.059514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:46:58.059478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:47:19.093921 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.093890    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"]
Jul 13 20:47:19.094451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.094076    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr" podUID="00854103-6356-4595-aecd-5e137d1f1ca1" containerName="maas-api" containerID="cri-o://34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4" gracePeriod=30
Jul 13 20:47:19.122917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.122887    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:47:19.342796 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.342772    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:47:19.391109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.391049    2584 generic.go:356] "Generic (PLEG): container finished" podID="00854103-6356-4595-aecd-5e137d1f1ca1" containerID="34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4" exitCode=0
Jul 13 20:47:19.391109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.391085    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr" event={"ID":"00854103-6356-4595-aecd-5e137d1f1ca1","Type":"ContainerDied","Data":"34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4"}
Jul 13 20:47:19.391241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.391086    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"
Jul 13 20:47:19.391241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.391125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr" event={"ID":"00854103-6356-4595-aecd-5e137d1f1ca1","Type":"ContainerDied","Data":"6b60fcb97c700fdfea731e5704d015fadef6384a0af8ba59abc95bbd6f9bdcc1"}
Jul 13 20:47:19.391241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.391158    2584 scope.go:117] "RemoveContainer" containerID="34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4"
Jul 13 20:47:19.398743 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.398719    2584 scope.go:117] "RemoveContainer" containerID="34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4"
Jul 13 20:47:19.399011 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:47:19.398992    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4\": container with ID starting with 34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4 not found: ID does not exist" containerID="34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4"
Jul 13 20:47:19.399081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.399020    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4"} err="failed to get container status \"34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4\": rpc error: code = NotFound desc = could not find container \"34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4\": container with ID starting with 34fa35b4ce88706e9b04b8dc749e276578d871b4dc0dc9364bb984090f58a9c4 not found: ID does not exist"
Jul 13 20:47:19.400223 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.400209    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00854103-6356-4595-aecd-5e137d1f1ca1-maas-api-tls\") pod \"00854103-6356-4595-aecd-5e137d1f1ca1\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") "
Jul 13 20:47:19.400269 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.400253    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkzwn\" (UniqueName: \"kubernetes.io/projected/00854103-6356-4595-aecd-5e137d1f1ca1-kube-api-access-kkzwn\") pod \"00854103-6356-4595-aecd-5e137d1f1ca1\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") "
Jul 13 20:47:19.400302 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.400284    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00854103-6356-4595-aecd-5e137d1f1ca1-openshift-service-ca\") pod \"00854103-6356-4595-aecd-5e137d1f1ca1\" (UID: \"00854103-6356-4595-aecd-5e137d1f1ca1\") "
Jul 13 20:47:19.400677 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.400652    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00854103-6356-4595-aecd-5e137d1f1ca1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00854103-6356-4595-aecd-5e137d1f1ca1" (UID: "00854103-6356-4595-aecd-5e137d1f1ca1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:47:19.402355 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.402331    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00854103-6356-4595-aecd-5e137d1f1ca1-kube-api-access-kkzwn" (OuterVolumeSpecName: "kube-api-access-kkzwn") pod "00854103-6356-4595-aecd-5e137d1f1ca1" (UID: "00854103-6356-4595-aecd-5e137d1f1ca1"). InnerVolumeSpecName "kube-api-access-kkzwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:47:19.402355 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.402340    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00854103-6356-4595-aecd-5e137d1f1ca1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00854103-6356-4595-aecd-5e137d1f1ca1" (UID: "00854103-6356-4595-aecd-5e137d1f1ca1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:47:19.501100 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.501070    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00854103-6356-4595-aecd-5e137d1f1ca1-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:19.501100 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.501100    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkzwn\" (UniqueName: \"kubernetes.io/projected/00854103-6356-4595-aecd-5e137d1f1ca1-kube-api-access-kkzwn\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:19.501100 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.501110    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00854103-6356-4595-aecd-5e137d1f1ca1-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:19.714286 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.712744    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"]
Jul 13 20:47:19.719230 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:19.719207    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d387c72d-65fb8bbd94-ntzzr"]
Jul 13 20:47:20.637100 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:20.637064    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00854103-6356-4595-aecd-5e137d1f1ca1" path="/var/lib/kubelet/pods/00854103-6356-4595-aecd-5e137d1f1ca1/volumes"
Jul 13 20:47:23.702149 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:23.702101    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:47:26.677279 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:26.677250    2584 scope.go:117] "RemoveContainer" containerID="a00bd182911f6c84392d658d81b93f92018dc2a63f970b365d03c8566572130b"
Jul 13 20:47:28.655869 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:28.655837    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"]
Jul 13 20:47:28.759146 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:28.759110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:47:43.675856 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:43.675822    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw" podUID="84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" containerName="main" containerID="cri-o://0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06" gracePeriod=45
Jul 13 20:47:43.921087 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:43.921063    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:47:44.075177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tls-certs\") pod \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") "
Jul 13 20:47:44.075177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075140    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98nkn\" (UniqueName: \"kubernetes.io/projected/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-kube-api-access-98nkn\") pod \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") "
Jul 13 20:47:44.075417 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075186    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-model-cache\") pod \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") "
Jul 13 20:47:44.075417 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075221    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-dshm\") pod \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") "
Jul 13 20:47:44.075417 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075238    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-home\") pod \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") "
Jul 13 20:47:44.075417 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075332    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tmp-dir\") pod \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\" (UID: \"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab\") "
Jul 13 20:47:44.075639 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-home" (OuterVolumeSpecName: "home") pod "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" (UID: "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:47:44.075639 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075533    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-model-cache" (OuterVolumeSpecName: "model-cache") pod "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" (UID: "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:47:44.075719 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075633    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" (UID: "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:47:44.075719 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075641    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:44.075719 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.075698    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:44.077503 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.077480    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-dshm" (OuterVolumeSpecName: "dshm") pod "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" (UID: "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:47:44.077574 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.077531    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" (UID: "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:47:44.077612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.077589    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-kube-api-access-98nkn" (OuterVolumeSpecName: "kube-api-access-98nkn") pod "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" (UID: "84d26ef5-adc2-4a84-8a2f-9dd0e6140cab"). InnerVolumeSpecName "kube-api-access-98nkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:47:44.176963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.176935    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:44.176963 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.176958    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:44.177096 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.176971    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:44.177096 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.176984    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98nkn\" (UniqueName: \"kubernetes.io/projected/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab-kube-api-access-98nkn\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:47:44.496916 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.496880    2584 generic.go:356] "Generic (PLEG): container finished" podID="84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" containerID="0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06" exitCode=0
Jul 13 20:47:44.497041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.496942    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"
Jul 13 20:47:44.497041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.496967    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw" event={"ID":"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab","Type":"ContainerDied","Data":"0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06"}
Jul 13 20:47:44.497041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.497017    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw" event={"ID":"84d26ef5-adc2-4a84-8a2f-9dd0e6140cab","Type":"ContainerDied","Data":"75f1d09c6ec39a18fe8eea52763f2780c9810845ac7c3fa5fef63e1c04cad150"}
Jul 13 20:47:44.497041 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.497040    2584 scope.go:117] "RemoveContainer" containerID="0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06"
Jul 13 20:47:44.505509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.505494    2584 scope.go:117] "RemoveContainer" containerID="0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06"
Jul 13 20:47:44.505742 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:47:44.505727    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06\": container with ID starting with 0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06 not found: ID does not exist" containerID="0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06"
Jul 13 20:47:44.505786 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.505748    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06"} err="failed to get container status \"0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06\": rpc error: code = NotFound desc = could not find container \"0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06\": container with ID starting with 0e7a7fbebe7cbf354960f518f99943f529d93fbe1ea5cf25f0cafe09f067dd06 not found: ID does not exist"
Jul 13 20:47:44.518040 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.518019    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"]
Jul 13 20:47:44.521377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.521353    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d387c72d/e2e-lifecycle-model-d387c72d-kserve-59cdb7c9ff-r4tgw"]
Jul 13 20:47:44.636328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:44.636305    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" path="/var/lib/kubelet/pods/84d26ef5-adc2-4a84-8a2f-9dd0e6140cab/volumes"
Jul 13 20:47:58.153220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:58.153183    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:47:58.206626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:47:58.206585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:04.993920 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:04.993880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:05.699068 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:05.699029    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:11.273847 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:11.273806    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:16.852778 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:16.852732    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:17.001715 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:17.001681    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:25.926565 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:25.926528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:26.132355 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:26.132317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:26.698604 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:26.698581    2584 scope.go:117] "RemoveContainer" containerID="2628fad02005d6af948ec95ecbbda14bffc5bd2ca94002a6c68c02a62ae37344"
Jul 13 20:48:27.185450 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:27.185418    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:27.283860 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:27.283816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:27.393715 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:27.393684    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:28.111045 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:28.111009    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:28.203461 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:28.203430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:28.314328 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:28.314291    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:29.685693 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:29.685656    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:34.488154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:34.488110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:34.588471 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:34.588442    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:35.683812 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:35.683782    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:35.785158 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:35.785126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:36.372233 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:36.372204    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:42.084464 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:42.084423    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:42.185838 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:42.185803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:42.288507 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:42.288470    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:52.760518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:52.760427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:48:53.108056 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:48:53.108025    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:14.093769 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:14.093730    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:19.852424 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:19.852375    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:19.952479 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:19.952447    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:20.241057 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:20.240969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:20.523551 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:20.523462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:20.625215 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:20.625179    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:23.752983 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.752949    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:23.822435 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822377    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"]
Jul 13 20:49:23.822804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822790    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" containerName="main"
Jul 13 20:49:23.822849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822808    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" containerName="main"
Jul 13 20:49:23.822849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822834    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00854103-6356-4595-aecd-5e137d1f1ca1" containerName="maas-api"
Jul 13 20:49:23.822849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822843    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="00854103-6356-4595-aecd-5e137d1f1ca1" containerName="maas-api"
Jul 13 20:49:23.822946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822912    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d26ef5-adc2-4a84-8a2f-9dd0e6140cab" containerName="main"
Jul 13 20:49:23.822946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.822923    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="00854103-6356-4595-aecd-5e137d1f1ca1" containerName="maas-api"
Jul 13 20:49:23.826242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.826223    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.829160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.829126    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-abc63a-serving-cert\""
Jul 13 20:49:23.829160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.829138    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:49:23.829160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.829126    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:49:23.830262 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.830242    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:49:23.836204 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.836184    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"]
Jul 13 20:49:23.863778 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.863752    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:23.890180 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.890159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.890294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.890188    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1c785-d378-49f8-a296-7867b34bd0aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.890294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.890214    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4rxj\" (UniqueName: \"kubernetes.io/projected/aaa1c785-d378-49f8-a296-7867b34bd0aa-kube-api-access-p4rxj\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.945646 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.945619    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:23.990810 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.990786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1c785-d378-49f8-a296-7867b34bd0aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.990897 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.990818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4rxj\" (UniqueName: \"kubernetes.io/projected/aaa1c785-d378-49f8-a296-7867b34bd0aa-kube-api-access-p4rxj\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.990948 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.990897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:23.991015 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:49:23.991000    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-serving-cert: secret "maas-api-e2e-inf-a-abc63a-serving-cert" not found
Jul 13 20:49:23.991089 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:49:23.991078    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls podName:aaa1c785-d378-49f8-a296-7867b34bd0aa nodeName:}" failed. No retries permitted until 2026-07-13 20:49:24.491058225 +0000 UTC m=+2098.391760866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls") pod "maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" (UID: "aaa1c785-d378-49f8-a296-7867b34bd0aa") : secret "maas-api-e2e-inf-a-abc63a-serving-cert" not found
Jul 13 20:49:23.991509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.991492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1c785-d378-49f8-a296-7867b34bd0aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:24.000018 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:23.999996    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4rxj\" (UniqueName: \"kubernetes.io/projected/aaa1c785-d378-49f8-a296-7867b34bd0aa-kube-api-access-p4rxj\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:24.196744 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.196719    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:24.296034 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.295989    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:24.395526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.395494    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:24.494821 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.494760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:24.497427 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.497406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:24.737489 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.737463    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:24.866772 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.866745    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"]
Jul 13 20:49:24.868376 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:49:24.868349    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa1c785_d378_49f8_a296_7867b34bd0aa.slice/crio-a6340e916b09ab06b96e2c656c76fcf395bd8a7d4107759e5ca7721722bc2535 WatchSource:0}: Error finding container a6340e916b09ab06b96e2c656c76fcf395bd8a7d4107759e5ca7721722bc2535: Status 404 returned error can't find the container with id a6340e916b09ab06b96e2c656c76fcf395bd8a7d4107759e5ca7721722bc2535
Jul 13 20:49:24.894522 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:24.894485    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" event={"ID":"aaa1c785-d378-49f8-a296-7867b34bd0aa","Type":"ContainerStarted","Data":"a6340e916b09ab06b96e2c656c76fcf395bd8a7d4107759e5ca7721722bc2535"}
Jul 13 20:49:25.900514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:25.900475    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" event={"ID":"aaa1c785-d378-49f8-a296-7867b34bd0aa","Type":"ContainerStarted","Data":"b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8"}
Jul 13 20:49:25.900947 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:25.900607    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:25.918434 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:25.918370    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" podStartSLOduration=2.345740363 podStartE2EDuration="2.918356315s" totalImagesPullingTime="572.615952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:49:25.916601215 +0000 UTC m=+2099.817303876" watchObservedRunningTime="2026-07-13 20:49:25.918356315 +0000 UTC m=+2099.819058973"
Jul 13 20:49:26.717190 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:26.717154    2584 scope.go:117] "RemoveContainer" containerID="da5797f1d226fd01540702b02d7e7171d9c6da5ed539f095677e3e4b1064fd4e"
Jul 13 20:49:28.301374 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:28.301337    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:28.656032 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:28.655998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:28.966967 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:28.966889    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:29.141047 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:29.141010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:31.790727 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.790683    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:31.863793 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.863762    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:31.864817 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.864793    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"]
Jul 13 20:49:31.868120 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.868105    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:31.872613 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.872594    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a35e15-serving-cert\""
Jul 13 20:49:31.877977 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.877942    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"]
Jul 13 20:49:31.909483 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.909461    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:49:31.939602 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.939568    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:31.947827 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.947803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:31.947913 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.947843    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc37be8-25f7-48df-a65b-015103294b77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:31.947913 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:31.947884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jx9\" (UniqueName: \"kubernetes.io/projected/edc37be8-25f7-48df-a65b-015103294b77-kube-api-access-47jx9\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.048294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.048222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.048294 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.048256    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc37be8-25f7-48df-a65b-015103294b77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.048501 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.048342    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jx9\" (UniqueName: \"kubernetes.io/projected/edc37be8-25f7-48df-a65b-015103294b77-kube-api-access-47jx9\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.048501 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:49:32.048368    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-serving-cert: secret "maas-api-e2e-inf-b-a35e15-serving-cert" not found
Jul 13 20:49:32.048501 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:49:32.048441    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls podName:edc37be8-25f7-48df-a65b-015103294b77 nodeName:}" failed. No retries permitted until 2026-07-13 20:49:32.548426234 +0000 UTC m=+2106.449128877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls") pod "maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" (UID: "edc37be8-25f7-48df-a65b-015103294b77") : secret "maas-api-e2e-inf-b-a35e15-serving-cert" not found
Jul 13 20:49:32.048880 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.048863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc37be8-25f7-48df-a65b-015103294b77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.057072 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.057045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jx9\" (UniqueName: \"kubernetes.io/projected/edc37be8-25f7-48df-a65b-015103294b77-kube-api-access-47jx9\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.452610 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.452577    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:32.552581 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.552552    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.555084 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.555063    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls\") pod \"maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.620823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.620783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:32.780039 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.779978    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:32.910799 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.910777    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"]
Jul 13 20:49:32.912841 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:49:32.912813    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc37be8_25f7_48df_a65b_015103294b77.slice/crio-2b6b5ada4edd2f4a83b0ff68f355b821088cea6a2dca5bbfbaa939f49f8bfe50 WatchSource:0}: Error finding container 2b6b5ada4edd2f4a83b0ff68f355b821088cea6a2dca5bbfbaa939f49f8bfe50: Status 404 returned error can't find the container with id 2b6b5ada4edd2f4a83b0ff68f355b821088cea6a2dca5bbfbaa939f49f8bfe50
Jul 13 20:49:32.928109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:32.928079    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" event={"ID":"edc37be8-25f7-48df-a65b-015103294b77","Type":"ContainerStarted","Data":"2b6b5ada4edd2f4a83b0ff68f355b821088cea6a2dca5bbfbaa939f49f8bfe50"}
Jul 13 20:49:33.933899 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:33.933868    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" event={"ID":"edc37be8-25f7-48df-a65b-015103294b77","Type":"ContainerStarted","Data":"8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696"}
Jul 13 20:49:33.934280 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:33.933983    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:33.951466 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:33.951419    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" podStartSLOduration=2.595847639 podStartE2EDuration="2.951378873s" totalImagesPullingTime="355.531234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:49:33.950475016 +0000 UTC m=+2107.851177680" watchObservedRunningTime="2026-07-13 20:49:33.951378873 +0000 UTC m=+2107.852081534"
Jul 13 20:49:36.413301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:36.413256    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:39.943630 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:39.943602    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:49:45.006043 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:45.006010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:45.106052 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:45.106020    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:54.075473 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:54.075430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:49:54.174005 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:49:54.173951    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:50:09.894348 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.894317    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"]
Jul 13 20:50:09.897506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.897485    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:09.901859 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.901838    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-abc63a\"/\"kube-root-ca.crt\""
Jul 13 20:50:09.901972 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.901872    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-abc63a\"/\"test-model-abc63a-kserve-self-signed-certs\""
Jul 13 20:50:09.901972 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.901926    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-abc63a\"/\"default-dockercfg-2bh2n\""
Jul 13 20:50:09.901972 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.901872    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-abc63a\"/\"openshift-service-ca.crt\""
Jul 13 20:50:09.905767 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.905747    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"]
Jul 13 20:50:09.916241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.916212    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:50:09.917827 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.917808    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-tmp-dir\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:09.917898 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.917852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-model-cache\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:09.917898 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.917884    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47736e5-758c-4493-ab34-4a9c0498befb-tls-certs\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:09.917973 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.917902    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-dshm\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:09.917973 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.917924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-home\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:09.917973 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:09.917945    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xwf\" (UniqueName: \"kubernetes.io/projected/e47736e5-758c-4493-ab34-4a9c0498befb-kube-api-access-r9xwf\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.018649 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.018622    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47736e5-758c-4493-ab34-4a9c0498befb-tls-certs\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.018757 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.018658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-dshm\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.018757 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.018692    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-home\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.018757 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.018726    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xwf\" (UniqueName: \"kubernetes.io/projected/e47736e5-758c-4493-ab34-4a9c0498befb-kube-api-access-r9xwf\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.018893 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.018758    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-tmp-dir\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.018960 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.018914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-model-cache\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.019122 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.019090    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-home\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.019241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.019135    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-tmp-dir\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.019301 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.019239    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-model-cache\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.020952 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.020934    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-dshm\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.021289 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.021274    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47736e5-758c-4493-ab34-4a9c0498befb-tls-certs\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.026989 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.026961    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xwf\" (UniqueName: \"kubernetes.io/projected/e47736e5-758c-4493-ab34-4a9c0498befb-kube-api-access-r9xwf\") pod \"test-model-abc63a-kserve-7cdb6867d9-rfc7d\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") " pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.206832 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.206761    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:10.338987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:10.338891    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"]
Jul 13 20:50:10.341967 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:50:10.341935    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode47736e5_758c_4493_ab34_4a9c0498befb.slice/crio-a10634a957d4261c493087743944d5d51c3366382bf20561d2140e479a1c6f0a WatchSource:0}: Error finding container a10634a957d4261c493087743944d5d51c3366382bf20561d2140e479a1c6f0a: Status 404 returned error can't find the container with id a10634a957d4261c493087743944d5d51c3366382bf20561d2140e479a1c6f0a
Jul 13 20:50:11.080672 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:11.080627    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d" event={"ID":"e47736e5-758c-4493-ab34-4a9c0498befb","Type":"ContainerStarted","Data":"a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f"}
Jul 13 20:50:11.081035 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:11.080681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d" event={"ID":"e47736e5-758c-4493-ab34-4a9c0498befb","Type":"ContainerStarted","Data":"a10634a957d4261c493087743944d5d51c3366382bf20561d2140e479a1c6f0a"}
Jul 13 20:50:11.101141 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:11.101096    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d" podStartSLOduration=2.101082123 podStartE2EDuration="2.101082123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:50:11.100196628 +0000 UTC m=+2145.000899290" watchObservedRunningTime="2026-07-13 20:50:11.101082123 +0000 UTC m=+2145.001784785"
Jul 13 20:50:20.207449 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:20.207414    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:20.208024 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:20.207466    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:20.219491 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:20.219468    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:21.141362 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:21.141326    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:50:30.976636 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.976604    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"]
Jul 13 20:50:30.979774 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.979749    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:30.983671 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.983645    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a35e15\"/\"openshift-service-ca.crt\""
Jul 13 20:50:30.983804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.983689    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a35e15\"/\"default-dockercfg-dwzzm\""
Jul 13 20:50:30.983804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.983728    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a35e15\"/\"kube-root-ca.crt\""
Jul 13 20:50:30.983804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.983772    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a35e15\"/\"test-model-a35e15-kserve-self-signed-certs\""
Jul 13 20:50:30.989246 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:30.989223    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"]
Jul 13 20:50:31.007854 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.007833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:50:31.081067 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.081030    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-home\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.081174 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.081086    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-dshm\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.081174 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.081105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tmp-dir\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.081174 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.081144    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-model-cache\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.081174 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.081159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmhv8\" (UniqueName: \"kubernetes.io/projected/aad07edf-7b40-4017-ab6a-4b4be002b4b6-kube-api-access-zmhv8\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.081329 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.081204    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tls-certs\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182444    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-home\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182588 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-dshm\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182644 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tmp-dir\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182712 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-model-cache\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182712 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182661    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmhv8\" (UniqueName: \"kubernetes.io/projected/aad07edf-7b40-4017-ab6a-4b4be002b4b6-kube-api-access-zmhv8\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182712 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tls-certs\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182924 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-home\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182964    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tmp-dir\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.182981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.182974    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-model-cache\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.184778 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.184759    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-dshm\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.185077 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.185060    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tls-certs\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.191029 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.191006    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmhv8\" (UniqueName: \"kubernetes.io/projected/aad07edf-7b40-4017-ab6a-4b4be002b4b6-kube-api-access-zmhv8\") pod \"test-model-a35e15-kserve-548c7dc985-6gkl8\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") " pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.289705 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.289652    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:31.420488 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:31.420462    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"]
Jul 13 20:50:31.422847 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:50:31.422819    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad07edf_7b40_4017_ab6a_4b4be002b4b6.slice/crio-2f71d5128ede8393dc438df3b82e5f371af62cbb05de38c74c94a7bf34663e6f WatchSource:0}: Error finding container 2f71d5128ede8393dc438df3b82e5f371af62cbb05de38c74c94a7bf34663e6f: Status 404 returned error can't find the container with id 2f71d5128ede8393dc438df3b82e5f371af62cbb05de38c74c94a7bf34663e6f
Jul 13 20:50:32.175517 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:32.175481    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8" event={"ID":"aad07edf-7b40-4017-ab6a-4b4be002b4b6","Type":"ContainerStarted","Data":"b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5"}
Jul 13 20:50:32.175933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:32.175532    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8" event={"ID":"aad07edf-7b40-4017-ab6a-4b4be002b4b6","Type":"ContainerStarted","Data":"2f71d5128ede8393dc438df3b82e5f371af62cbb05de38c74c94a7bf34663e6f"}
Jul 13 20:50:32.192919 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:32.192878    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8" podStartSLOduration=2.192864662 podStartE2EDuration="2.192864662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:50:32.190266635 +0000 UTC m=+2166.090969296" watchObservedRunningTime="2026-07-13 20:50:32.192864662 +0000 UTC m=+2166.093567362"
Jul 13 20:50:41.290670 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:41.290631    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:41.291043 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:41.290695    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:41.303020 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:41.302993    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:42.226981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:42.226946    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:50:55.274321 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:55.274288    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"]
Jul 13 20:50:55.297047 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:55.297012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:50:56.292780 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:56.292738    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"]
Jul 13 20:50:56.312102 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:56.312072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:50:56.895158 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:56.895117    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:50:57.000151 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:50:57.000115    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:06.818861 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:06.818823    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"]
Jul 13 20:51:06.819360 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:06.819079    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" podUID="aaa1c785-d378-49f8-a296-7867b34bd0aa" containerName="maas-api" containerID="cri-o://b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8" gracePeriod=30
Jul 13 20:51:06.842673 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:06.842645    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:06.904659 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:06.904616    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" podUID="aaa1c785-d378-49f8-a296-7867b34bd0aa" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.38:8443/health\": dial tcp 10.133.0.38:8443: connect: connection refused"
Jul 13 20:51:07.260230 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.260204    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:51:07.325339 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.325298    2584 generic.go:356] "Generic (PLEG): container finished" podID="aaa1c785-d378-49f8-a296-7867b34bd0aa" containerID="b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8" exitCode=0
Jul 13 20:51:07.325506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.325348    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" event={"ID":"aaa1c785-d378-49f8-a296-7867b34bd0aa","Type":"ContainerDied","Data":"b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8"}
Jul 13 20:51:07.325506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.325362    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"
Jul 13 20:51:07.325506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.325448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l" event={"ID":"aaa1c785-d378-49f8-a296-7867b34bd0aa","Type":"ContainerDied","Data":"a6340e916b09ab06b96e2c656c76fcf395bd8a7d4107759e5ca7721722bc2535"}
Jul 13 20:51:07.325506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.325478    2584 scope.go:117] "RemoveContainer" containerID="b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8"
Jul 13 20:51:07.335559 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.335537    2584 scope.go:117] "RemoveContainer" containerID="b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8"
Jul 13 20:51:07.335838 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:51:07.335818    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8\": container with ID starting with b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8 not found: ID does not exist" containerID="b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8"
Jul 13 20:51:07.335904 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.335850    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8"} err="failed to get container status \"b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8\": rpc error: code = NotFound desc = could not find container \"b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8\": container with ID starting with b239c19cb33895fe0e944ff523dcc307272afd42c06d99565af9a6a9bcfc01d8 not found: ID does not exist"
Jul 13 20:51:07.354302 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.354249    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1c785-d378-49f8-a296-7867b34bd0aa-openshift-service-ca\") pod \"aaa1c785-d378-49f8-a296-7867b34bd0aa\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") "
Jul 13 20:51:07.354302 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.354280    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4rxj\" (UniqueName: \"kubernetes.io/projected/aaa1c785-d378-49f8-a296-7867b34bd0aa-kube-api-access-p4rxj\") pod \"aaa1c785-d378-49f8-a296-7867b34bd0aa\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") "
Jul 13 20:51:07.354438 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.354325    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls\") pod \"aaa1c785-d378-49f8-a296-7867b34bd0aa\" (UID: \"aaa1c785-d378-49f8-a296-7867b34bd0aa\") "
Jul 13 20:51:07.354669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.354645    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa1c785-d378-49f8-a296-7867b34bd0aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aaa1c785-d378-49f8-a296-7867b34bd0aa" (UID: "aaa1c785-d378-49f8-a296-7867b34bd0aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:51:07.356492 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.356459    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa1c785-d378-49f8-a296-7867b34bd0aa-kube-api-access-p4rxj" (OuterVolumeSpecName: "kube-api-access-p4rxj") pod "aaa1c785-d378-49f8-a296-7867b34bd0aa" (UID: "aaa1c785-d378-49f8-a296-7867b34bd0aa"). InnerVolumeSpecName "kube-api-access-p4rxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:51:07.356612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.356544    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aaa1c785-d378-49f8-a296-7867b34bd0aa" (UID: "aaa1c785-d378-49f8-a296-7867b34bd0aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:51:07.455750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.455729    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaa1c785-d378-49f8-a296-7867b34bd0aa-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:07.455750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.455749    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4rxj\" (UniqueName: \"kubernetes.io/projected/aaa1c785-d378-49f8-a296-7867b34bd0aa-kube-api-access-p4rxj\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:07.455875 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.455759    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaa1c785-d378-49f8-a296-7867b34bd0aa-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:07.647015 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.646987    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"]
Jul 13 20:51:07.649922 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:07.649900    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abc63a-86cd8c6c56-7x49l"]
Jul 13 20:51:08.637129 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:08.637095    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaa1c785-d378-49f8-a296-7867b34bd0aa" path="/var/lib/kubelet/pods/aaa1c785-d378-49f8-a296-7867b34bd0aa/volumes"
Jul 13 20:51:10.294616 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.294578    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d" podUID="e47736e5-758c-4493-ab34-4a9c0498befb" containerName="main" containerID="cri-o://a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f" gracePeriod=45
Jul 13 20:51:10.538892 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.538871    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:51:10.676506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676484    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-model-cache\") pod \"e47736e5-758c-4493-ab34-4a9c0498befb\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") "
Jul 13 20:51:10.676661 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676528    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9xwf\" (UniqueName: \"kubernetes.io/projected/e47736e5-758c-4493-ab34-4a9c0498befb-kube-api-access-r9xwf\") pod \"e47736e5-758c-4493-ab34-4a9c0498befb\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") "
Jul 13 20:51:10.676661 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676553    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47736e5-758c-4493-ab34-4a9c0498befb-tls-certs\") pod \"e47736e5-758c-4493-ab34-4a9c0498befb\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") "
Jul 13 20:51:10.676750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676676    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-home\") pod \"e47736e5-758c-4493-ab34-4a9c0498befb\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") "
Jul 13 20:51:10.676750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676701    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-tmp-dir\") pod \"e47736e5-758c-4493-ab34-4a9c0498befb\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") "
Jul 13 20:51:10.676750 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676726    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-dshm\") pod \"e47736e5-758c-4493-ab34-4a9c0498befb\" (UID: \"e47736e5-758c-4493-ab34-4a9c0498befb\") "
Jul 13 20:51:10.676905 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676784    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e47736e5-758c-4493-ab34-4a9c0498befb" (UID: "e47736e5-758c-4493-ab34-4a9c0498befb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:10.676991 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676969    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-home" (OuterVolumeSpecName: "home") pod "e47736e5-758c-4493-ab34-4a9c0498befb" (UID: "e47736e5-758c-4493-ab34-4a9c0498befb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:10.677044 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676983    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e47736e5-758c-4493-ab34-4a9c0498befb" (UID: "e47736e5-758c-4493-ab34-4a9c0498befb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:10.677044 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.676975    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:10.678848 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.678819    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-dshm" (OuterVolumeSpecName: "dshm") pod "e47736e5-758c-4493-ab34-4a9c0498befb" (UID: "e47736e5-758c-4493-ab34-4a9c0498befb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:10.678945 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.678888    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47736e5-758c-4493-ab34-4a9c0498befb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e47736e5-758c-4493-ab34-4a9c0498befb" (UID: "e47736e5-758c-4493-ab34-4a9c0498befb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:51:10.678945 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.678930    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47736e5-758c-4493-ab34-4a9c0498befb-kube-api-access-r9xwf" (OuterVolumeSpecName: "kube-api-access-r9xwf") pod "e47736e5-758c-4493-ab34-4a9c0498befb" (UID: "e47736e5-758c-4493-ab34-4a9c0498befb"). InnerVolumeSpecName "kube-api-access-r9xwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:51:10.777342 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.777308    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e47736e5-758c-4493-ab34-4a9c0498befb-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:10.777342 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.777339    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:10.777342 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.777348    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:10.777520 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.777357    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e47736e5-758c-4493-ab34-4a9c0498befb-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:10.777520 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:10.777364    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9xwf\" (UniqueName: \"kubernetes.io/projected/e47736e5-758c-4493-ab34-4a9c0498befb-kube-api-access-r9xwf\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:11.313745 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.313707    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8" podUID="aad07edf-7b40-4017-ab6a-4b4be002b4b6" containerName="main" containerID="cri-o://b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5" gracePeriod=45
Jul 13 20:51:11.343789 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.343762    2584 generic.go:356] "Generic (PLEG): container finished" podID="e47736e5-758c-4493-ab34-4a9c0498befb" containerID="a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f" exitCode=0
Jul 13 20:51:11.343900 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.343805    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"
Jul 13 20:51:11.343900 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.343847    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d" event={"ID":"e47736e5-758c-4493-ab34-4a9c0498befb","Type":"ContainerDied","Data":"a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f"}
Jul 13 20:51:11.343900 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.343889    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d" event={"ID":"e47736e5-758c-4493-ab34-4a9c0498befb","Type":"ContainerDied","Data":"a10634a957d4261c493087743944d5d51c3366382bf20561d2140e479a1c6f0a"}
Jul 13 20:51:11.344034 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.343906    2584 scope.go:117] "RemoveContainer" containerID="a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f"
Jul 13 20:51:11.391118 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.391095    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"]
Jul 13 20:51:11.397460 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.395590    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-abc63a/test-model-abc63a-kserve-7cdb6867d9-rfc7d"]
Jul 13 20:51:11.425509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.425486    2584 scope.go:117] "RemoveContainer" containerID="a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f"
Jul 13 20:51:11.425808 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:51:11.425790    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f\": container with ID starting with a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f not found: ID does not exist" containerID="a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f"
Jul 13 20:51:11.425852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.425814    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f"} err="failed to get container status \"a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f\": rpc error: code = NotFound desc = could not find container \"a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f\": container with ID starting with a31055004f778f6c937337a0f962c9ff46c6f52771ef355c7a9193fd1954ef1f not found: ID does not exist"
Jul 13 20:51:11.523263 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.523231    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:11.561684 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.561662    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:51:11.684108 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684084    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmhv8\" (UniqueName: \"kubernetes.io/projected/aad07edf-7b40-4017-ab6a-4b4be002b4b6-kube-api-access-zmhv8\") pod \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") "
Jul 13 20:51:11.684242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684117    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-dshm\") pod \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") "
Jul 13 20:51:11.684242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684134    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-model-cache\") pod \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") "
Jul 13 20:51:11.684242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684155    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tls-certs\") pod \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") "
Jul 13 20:51:11.684242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684179    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-home\") pod \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") "
Jul 13 20:51:11.684242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684236    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tmp-dir\") pod \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\" (UID: \"aad07edf-7b40-4017-ab6a-4b4be002b4b6\") "
Jul 13 20:51:11.684549 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684446    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "aad07edf-7b40-4017-ab6a-4b4be002b4b6" (UID: "aad07edf-7b40-4017-ab6a-4b4be002b4b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:11.684549 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684522    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-home" (OuterVolumeSpecName: "home") pod "aad07edf-7b40-4017-ab6a-4b4be002b4b6" (UID: "aad07edf-7b40-4017-ab6a-4b4be002b4b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:11.684646 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.684597    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aad07edf-7b40-4017-ab6a-4b4be002b4b6" (UID: "aad07edf-7b40-4017-ab6a-4b4be002b4b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:11.686505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.686473    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-dshm" (OuterVolumeSpecName: "dshm") pod "aad07edf-7b40-4017-ab6a-4b4be002b4b6" (UID: "aad07edf-7b40-4017-ab6a-4b4be002b4b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:51:11.686592 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.686537    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad07edf-7b40-4017-ab6a-4b4be002b4b6-kube-api-access-zmhv8" (OuterVolumeSpecName: "kube-api-access-zmhv8") pod "aad07edf-7b40-4017-ab6a-4b4be002b4b6" (UID: "aad07edf-7b40-4017-ab6a-4b4be002b4b6"). InnerVolumeSpecName "kube-api-access-zmhv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:51:11.686592 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.686581    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aad07edf-7b40-4017-ab6a-4b4be002b4b6" (UID: "aad07edf-7b40-4017-ab6a-4b4be002b4b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:51:11.784867 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.784844    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmhv8\" (UniqueName: \"kubernetes.io/projected/aad07edf-7b40-4017-ab6a-4b4be002b4b6-kube-api-access-zmhv8\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:11.784867 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.784864    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:11.784987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.784875    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:11.784987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.784884    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:11.784987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.784891    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:11.784987 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:11.784917    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad07edf-7b40-4017-ab6a-4b4be002b4b6-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:12.349758 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.349726    2584 generic.go:356] "Generic (PLEG): container finished" podID="aad07edf-7b40-4017-ab6a-4b4be002b4b6" containerID="b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5" exitCode=0
Jul 13 20:51:12.350228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.349767    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"
Jul 13 20:51:12.350228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.349802    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8" event={"ID":"aad07edf-7b40-4017-ab6a-4b4be002b4b6","Type":"ContainerDied","Data":"b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5"}
Jul 13 20:51:12.350228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.349845    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8" event={"ID":"aad07edf-7b40-4017-ab6a-4b4be002b4b6","Type":"ContainerDied","Data":"2f71d5128ede8393dc438df3b82e5f371af62cbb05de38c74c94a7bf34663e6f"}
Jul 13 20:51:12.350228 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.349866    2584 scope.go:117] "RemoveContainer" containerID="b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5"
Jul 13 20:51:12.358504 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.358487    2584 scope.go:117] "RemoveContainer" containerID="b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5"
Jul 13 20:51:12.358763 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:51:12.358742    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5\": container with ID starting with b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5 not found: ID does not exist" containerID="b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5"
Jul 13 20:51:12.358839 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.358775    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5"} err="failed to get container status \"b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5\": rpc error: code = NotFound desc = could not find container \"b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5\": container with ID starting with b17af55d1bbef6baffdda5144fd8b9bd6109b36acf6533b4f434bc9e17f91aa5 not found: ID does not exist"
Jul 13 20:51:12.369126 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.369104    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"]
Jul 13 20:51:12.375257 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.375237    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a35e15/test-model-a35e15-kserve-548c7dc985-6gkl8"]
Jul 13 20:51:12.637191 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.637132    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad07edf-7b40-4017-ab6a-4b4be002b4b6" path="/var/lib/kubelet/pods/aad07edf-7b40-4017-ab6a-4b4be002b4b6/volumes"
Jul 13 20:51:12.637514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:12.637500    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e47736e5-758c-4493-ab34-4a9c0498befb" path="/var/lib/kubelet/pods/e47736e5-758c-4493-ab34-4a9c0498befb/volumes"
Jul 13 20:51:17.294122 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:17.294081    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:17.386902 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:17.386871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:22.220618 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:22.220580    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:22.458708 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:22.458679    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:23.281053 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:23.281017    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:23.384801 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:23.384765    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:32.971242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:32.971212    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"]
Jul 13 20:51:32.971674 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:32.971437    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" podUID="edc37be8-25f7-48df-a65b-015103294b77" containerName="maas-api" containerID="cri-o://8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696" gracePeriod=30
Jul 13 20:51:32.993784 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:32.993755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:33.208764 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.208734    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:51:33.337608 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.337586    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls\") pod \"edc37be8-25f7-48df-a65b-015103294b77\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") "
Jul 13 20:51:33.337725 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.337616    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47jx9\" (UniqueName: \"kubernetes.io/projected/edc37be8-25f7-48df-a65b-015103294b77-kube-api-access-47jx9\") pod \"edc37be8-25f7-48df-a65b-015103294b77\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") "
Jul 13 20:51:33.337725 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.337642    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc37be8-25f7-48df-a65b-015103294b77-openshift-service-ca\") pod \"edc37be8-25f7-48df-a65b-015103294b77\" (UID: \"edc37be8-25f7-48df-a65b-015103294b77\") "
Jul 13 20:51:33.338053 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.338024    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc37be8-25f7-48df-a65b-015103294b77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edc37be8-25f7-48df-a65b-015103294b77" (UID: "edc37be8-25f7-48df-a65b-015103294b77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:51:33.339814 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.339789    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc37be8-25f7-48df-a65b-015103294b77-kube-api-access-47jx9" (OuterVolumeSpecName: "kube-api-access-47jx9") pod "edc37be8-25f7-48df-a65b-015103294b77" (UID: "edc37be8-25f7-48df-a65b-015103294b77"). InnerVolumeSpecName "kube-api-access-47jx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:51:33.339896 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.339870    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edc37be8-25f7-48df-a65b-015103294b77" (UID: "edc37be8-25f7-48df-a65b-015103294b77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:51:33.438282 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.438254    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc37be8-25f7-48df-a65b-015103294b77-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:33.438282 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.438280    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jx9\" (UniqueName: \"kubernetes.io/projected/edc37be8-25f7-48df-a65b-015103294b77-kube-api-access-47jx9\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:33.438468 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.438290    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc37be8-25f7-48df-a65b-015103294b77-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:51:33.440118 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.440091    2584 generic.go:356] "Generic (PLEG): container finished" podID="edc37be8-25f7-48df-a65b-015103294b77" containerID="8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696" exitCode=0
Jul 13 20:51:33.440215 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.440137    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"
Jul 13 20:51:33.440215 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.440153    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" event={"ID":"edc37be8-25f7-48df-a65b-015103294b77","Type":"ContainerDied","Data":"8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696"}
Jul 13 20:51:33.440215 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.440201    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7" event={"ID":"edc37be8-25f7-48df-a65b-015103294b77","Type":"ContainerDied","Data":"2b6b5ada4edd2f4a83b0ff68f355b821088cea6a2dca5bbfbaa939f49f8bfe50"}
Jul 13 20:51:33.440315 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.440223    2584 scope.go:117] "RemoveContainer" containerID="8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696"
Jul 13 20:51:33.448473 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.448457    2584 scope.go:117] "RemoveContainer" containerID="8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696"
Jul 13 20:51:33.448764 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:51:33.448745    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696\": container with ID starting with 8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696 not found: ID does not exist" containerID="8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696"
Jul 13 20:51:33.448819 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.448772    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696"} err="failed to get container status \"8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696\": rpc error: code = NotFound desc = could not find container \"8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696\": container with ID starting with 8868fc81cadd12da531fb28a4ea618e927762fe5f457fab10a0264232a576696 not found: ID does not exist"
Jul 13 20:51:33.460759 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.460739    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"]
Jul 13 20:51:33.464081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:33.464061    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a35e15-cb6b8cffc-nbrc7"]
Jul 13 20:51:34.636649 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:34.636613    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc37be8-25f7-48df-a65b-015103294b77" path="/var/lib/kubelet/pods/edc37be8-25f7-48df-a65b-015103294b77/volumes"
Jul 13 20:51:37.662428 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:37.662355    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:43.705984 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:43.705952    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:43.805452 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:43.805415    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:48.358784 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:48.358674    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:48.605334 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:48.605296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:48.963074 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:48.963036    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:49.662691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:49.662648    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:49.910855 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:49.910820    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:52.608940 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.608902    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:52.677818 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.677782    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"]
Jul 13 20:51:52.678154 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678135    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc37be8-25f7-48df-a65b-015103294b77" containerName="maas-api"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678158    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc37be8-25f7-48df-a65b-015103294b77" containerName="maas-api"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678182    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47736e5-758c-4493-ab34-4a9c0498befb" containerName="main"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678190    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47736e5-758c-4493-ab34-4a9c0498befb" containerName="main"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678202    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad07edf-7b40-4017-ab6a-4b4be002b4b6" containerName="main"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678211    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad07edf-7b40-4017-ab6a-4b4be002b4b6" containerName="main"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678224    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaa1c785-d378-49f8-a296-7867b34bd0aa" containerName="maas-api"
Jul 13 20:51:52.678244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678232    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaa1c785-d378-49f8-a296-7867b34bd0aa" containerName="maas-api"
Jul 13 20:51:52.678612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678299    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47736e5-758c-4493-ab34-4a9c0498befb" containerName="main"
Jul 13 20:51:52.678612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678314    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc37be8-25f7-48df-a65b-015103294b77" containerName="maas-api"
Jul 13 20:51:52.678612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678323    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad07edf-7b40-4017-ab6a-4b4be002b4b6" containerName="main"
Jul 13 20:51:52.678612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.678332    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaa1c785-d378-49f8-a296-7867b34bd0aa" containerName="maas-api"
Jul 13 20:51:52.682526 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.682504    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.686577 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.686555    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ad7818-serving-cert\""
Jul 13 20:51:52.686725 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.686565    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:51:52.686839 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.686712    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:51:52.687562 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.687542    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:51:52.691902 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.691878    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"]
Jul 13 20:51:52.695125 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.695097    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:52.761892 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.761870    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:52.780043 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.780022    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.780121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.780052    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a0b8f5-4532-49ac-9b46-8230567b96eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.780121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.780108    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4bnh\" (UniqueName: \"kubernetes.io/projected/05a0b8f5-4532-49ac-9b46-8230567b96eb-kube-api-access-t4bnh\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.880669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.880591    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.880669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.880637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a0b8f5-4532-49ac-9b46-8230567b96eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.880854 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.880692    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4bnh\" (UniqueName: \"kubernetes.io/projected/05a0b8f5-4532-49ac-9b46-8230567b96eb-kube-api-access-t4bnh\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.880854 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:51:52.880739    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-serving-cert: secret "maas-api-e2e-api-a-ad7818-serving-cert" not found
Jul 13 20:51:52.880854 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:51:52.880814    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls podName:05a0b8f5-4532-49ac-9b46-8230567b96eb nodeName:}" failed. No retries permitted until 2026-07-13 20:51:53.380797348 +0000 UTC m=+2247.281499988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls") pod "maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" (UID: "05a0b8f5-4532-49ac-9b46-8230567b96eb") : secret "maas-api-e2e-api-a-ad7818-serving-cert" not found
Jul 13 20:51:52.881317 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.881298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a0b8f5-4532-49ac-9b46-8230567b96eb-openshift-service-ca\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:52.889575 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:52.889547    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4bnh\" (UniqueName: \"kubernetes.io/projected/05a0b8f5-4532-49ac-9b46-8230567b96eb-kube-api-access-t4bnh\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:53.029522 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.029481    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:53.247680 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.247602    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:53.348411 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.348373    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:53.385266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.385241    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:53.387713 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.387691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls\") pod \"maas-api-e2e-api-a-ad7818-c99955c76-8h7fx\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:53.594297 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.594272    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:53.721913 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.721883    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"]
Jul 13 20:51:53.725511 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:51:53.725477    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a0b8f5_4532_49ac_9b46_8230567b96eb.slice/crio-8f5e5c7bf08521e14a3f454327a2012582c202e4d997f54f4d1769bb0fa22858 WatchSource:0}: Error finding container 8f5e5c7bf08521e14a3f454327a2012582c202e4d997f54f4d1769bb0fa22858: Status 404 returned error can't find the container with id 8f5e5c7bf08521e14a3f454327a2012582c202e4d997f54f4d1769bb0fa22858
Jul 13 20:51:53.727239 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:53.727219    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:51:54.525076 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:54.525003    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" event={"ID":"05a0b8f5-4532-49ac-9b46-8230567b96eb","Type":"ContainerStarted","Data":"4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4"}
Jul 13 20:51:54.525076 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:54.525055    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" event={"ID":"05a0b8f5-4532-49ac-9b46-8230567b96eb","Type":"ContainerStarted","Data":"8f5e5c7bf08521e14a3f454327a2012582c202e4d997f54f4d1769bb0fa22858"}
Jul 13 20:51:54.525269 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:54.525118    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:51:54.540970 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:54.540926    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" podStartSLOduration=2.032113715 podStartE2EDuration="2.540912228s" totalImagesPullingTime="508.798513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:51:54.540169338 +0000 UTC m=+2248.440872001" watchObservedRunningTime="2026-07-13 20:51:54.540912228 +0000 UTC m=+2248.441614889"
Jul 13 20:51:57.119828 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:57.119783    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:57.578472 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:57.578345    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:57.881125 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:57.881088    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:51:58.165969 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:51:58.165892    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:00.534277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.534244    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:52:00.686539 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.686509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:00.737034 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.737004    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"]
Jul 13 20:52:00.740031 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.740015    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.742768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.742746    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-590516-serving-cert\""
Jul 13 20:52:00.749089 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.749066    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"]
Jul 13 20:52:00.771322 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.771296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:00.843608 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.843579    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.843771 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.843621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6db7a9-78d2-4603-bc99-2f6c3865d3da-openshift-service-ca\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.843771 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.843686    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrjp\" (UniqueName: \"kubernetes.io/projected/df6db7a9-78d2-4603-bc99-2f6c3865d3da-kube-api-access-vhrjp\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.848035 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.848008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:00.944508 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.944477    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.944648 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.944520    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6db7a9-78d2-4603-bc99-2f6c3865d3da-openshift-service-ca\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.944648 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.944583    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrjp\" (UniqueName: \"kubernetes.io/projected/df6db7a9-78d2-4603-bc99-2f6c3865d3da-kube-api-access-vhrjp\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.944648 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:52:00.944624    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-590516-serving-cert: secret "maas-api-e2e-api-b-590516-serving-cert" not found
Jul 13 20:52:00.944828 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:52:00.944678    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls podName:df6db7a9-78d2-4603-bc99-2f6c3865d3da nodeName:}" failed. No retries permitted until 2026-07-13 20:52:01.444663333 +0000 UTC m=+2255.345365972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls") pod "maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" (UID: "df6db7a9-78d2-4603-bc99-2f6c3865d3da") : secret "maas-api-e2e-api-b-590516-serving-cert" not found
Jul 13 20:52:00.945171 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.945153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6db7a9-78d2-4603-bc99-2f6c3865d3da-openshift-service-ca\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:00.953635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:00.953612    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrjp\" (UniqueName: \"kubernetes.io/projected/df6db7a9-78d2-4603-bc99-2f6c3865d3da-kube-api-access-vhrjp\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:01.219677 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.219588    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:01.318672 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.318621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:01.419170 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.419128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:01.449484 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.449452    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:01.452081 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.452053    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls\") pod \"maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:01.652121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.652087    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:01.783669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:01.783644    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"]
Jul 13 20:52:01.785828 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:52:01.785795    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6db7a9_78d2_4603_bc99_2f6c3865d3da.slice/crio-50a37510496686408c444857fac72a94f05922a352ce0eb171d4aa5bf1ea31c8 WatchSource:0}: Error finding container 50a37510496686408c444857fac72a94f05922a352ce0eb171d4aa5bf1ea31c8: Status 404 returned error can't find the container with id 50a37510496686408c444857fac72a94f05922a352ce0eb171d4aa5bf1ea31c8
Jul 13 20:52:02.559672 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:02.559589    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" event={"ID":"df6db7a9-78d2-4603-bc99-2f6c3865d3da","Type":"ContainerStarted","Data":"f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a"}
Jul 13 20:52:02.559672 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:02.559651    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" event={"ID":"df6db7a9-78d2-4603-bc99-2f6c3865d3da","Type":"ContainerStarted","Data":"50a37510496686408c444857fac72a94f05922a352ce0eb171d4aa5bf1ea31c8"}
Jul 13 20:52:02.559842 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:02.559677    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:02.575284 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:02.575244    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" podStartSLOduration=2.185626498 podStartE2EDuration="2.575232077s" totalImagesPullingTime="389.605579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:52:02.574965488 +0000 UTC m=+2256.475668149" watchObservedRunningTime="2026-07-13 20:52:02.575232077 +0000 UTC m=+2256.475934739"
Jul 13 20:52:05.168990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:05.168953    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:08.568933 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:08.568906    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:13.692749 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:13.692717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:13.789882 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:13.789851    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:22.814673 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:22.814626    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:23.403758 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.403724    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"]
Jul 13 20:52:23.403983 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.403959    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" podUID="05a0b8f5-4532-49ac-9b46-8230567b96eb" containerName="maas-api" containerID="cri-o://4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4" gracePeriod=30
Jul 13 20:52:23.642859 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.642836    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:52:23.645765 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.645742    2584 generic.go:356] "Generic (PLEG): container finished" podID="05a0b8f5-4532-49ac-9b46-8230567b96eb" containerID="4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4" exitCode=0
Jul 13 20:52:23.645883 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.645779    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"
Jul 13 20:52:23.645883 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.645783    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" event={"ID":"05a0b8f5-4532-49ac-9b46-8230567b96eb","Type":"ContainerDied","Data":"4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4"}
Jul 13 20:52:23.645883 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.645836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx" event={"ID":"05a0b8f5-4532-49ac-9b46-8230567b96eb","Type":"ContainerDied","Data":"8f5e5c7bf08521e14a3f454327a2012582c202e4d997f54f4d1769bb0fa22858"}
Jul 13 20:52:23.645883 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.645868    2584 scope.go:117] "RemoveContainer" containerID="4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4"
Jul 13 20:52:23.653263 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.653248    2584 scope.go:117] "RemoveContainer" containerID="4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4"
Jul 13 20:52:23.653536 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:52:23.653518    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4\": container with ID starting with 4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4 not found: ID does not exist" containerID="4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4"
Jul 13 20:52:23.653591 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.653544    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4"} err="failed to get container status \"4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4\": rpc error: code = NotFound desc = could not find container \"4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4\": container with ID starting with 4f1bc396f13a94233df4b10bc3434086605638ee54cf7096c93706d789f407c4 not found: ID does not exist"
Jul 13 20:52:23.803603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.803543    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4bnh\" (UniqueName: \"kubernetes.io/projected/05a0b8f5-4532-49ac-9b46-8230567b96eb-kube-api-access-t4bnh\") pod \"05a0b8f5-4532-49ac-9b46-8230567b96eb\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") "
Jul 13 20:52:23.803603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.803574    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls\") pod \"05a0b8f5-4532-49ac-9b46-8230567b96eb\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") "
Jul 13 20:52:23.803766 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.803626    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a0b8f5-4532-49ac-9b46-8230567b96eb-openshift-service-ca\") pod \"05a0b8f5-4532-49ac-9b46-8230567b96eb\" (UID: \"05a0b8f5-4532-49ac-9b46-8230567b96eb\") "
Jul 13 20:52:23.803984 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.803960    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a0b8f5-4532-49ac-9b46-8230567b96eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05a0b8f5-4532-49ac-9b46-8230567b96eb" (UID: "05a0b8f5-4532-49ac-9b46-8230567b96eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:52:23.805854 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.805832    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05a0b8f5-4532-49ac-9b46-8230567b96eb" (UID: "05a0b8f5-4532-49ac-9b46-8230567b96eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:52:23.805947 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.805921    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a0b8f5-4532-49ac-9b46-8230567b96eb-kube-api-access-t4bnh" (OuterVolumeSpecName: "kube-api-access-t4bnh") pod "05a0b8f5-4532-49ac-9b46-8230567b96eb" (UID: "05a0b8f5-4532-49ac-9b46-8230567b96eb"). InnerVolumeSpecName "kube-api-access-t4bnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:52:23.909834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.905284    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05a0b8f5-4532-49ac-9b46-8230567b96eb-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:23.909834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.905331    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4bnh\" (UniqueName: \"kubernetes.io/projected/05a0b8f5-4532-49ac-9b46-8230567b96eb-kube-api-access-t4bnh\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:23.909834 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.905345    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05a0b8f5-4532-49ac-9b46-8230567b96eb-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:23.965938 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.965910    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"]
Jul 13 20:52:23.969210 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:23.969188    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ad7818-c99955c76-8h7fx"]
Jul 13 20:52:24.080890 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:24.080827    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:24.637437 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:24.637380    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a0b8f5-4532-49ac-9b46-8230567b96eb" path="/var/lib/kubelet/pods/05a0b8f5-4532-49ac-9b46-8230567b96eb/volumes"
Jul 13 20:52:28.301124 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:28.301087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:34.090991 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:34.090955    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:34.256035 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:34.256006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:38.997977 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:38.997944    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:39.226363 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.226327    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:39.348226 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.348197    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"]
Jul 13 20:52:39.348482 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.348470    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a0b8f5-4532-49ac-9b46-8230567b96eb" containerName="maas-api"
Jul 13 20:52:39.348525 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.348484    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a0b8f5-4532-49ac-9b46-8230567b96eb" containerName="maas-api"
Jul 13 20:52:39.348574 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.348540    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a0b8f5-4532-49ac-9b46-8230567b96eb" containerName="maas-api"
Jul 13 20:52:39.351215 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.351198    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:39.353786 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.353769    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:52:39.358826 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.358806    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"]
Jul 13 20:52:39.519706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.519665    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c816668-21a1-4fbd-8eb9-23f6c03317e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-590516-fx6r2\" (UID: \"8c816668-21a1-4fbd-8eb9-23f6c03317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:39.620569 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.620474    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c816668-21a1-4fbd-8eb9-23f6c03317e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-590516-fx6r2\" (UID: \"8c816668-21a1-4fbd-8eb9-23f6c03317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:39.621126 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.621105    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c816668-21a1-4fbd-8eb9-23f6c03317e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-590516-fx6r2\" (UID: \"8c816668-21a1-4fbd-8eb9-23f6c03317e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:39.662178 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.662148    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:39.791139 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:39.791112    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"]
Jul 13 20:52:39.793594 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:52:39.793570    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c816668_21a1_4fbd_8eb9_23f6c03317e7.slice/crio-a3fc17ecd052a1d455970e2daebb507f7c25b82e1a4937267e4a66a0600265a1 WatchSource:0}: Error finding container a3fc17ecd052a1d455970e2daebb507f7c25b82e1a4937267e4a66a0600265a1: Status 404 returned error can't find the container with id a3fc17ecd052a1d455970e2daebb507f7c25b82e1a4937267e4a66a0600265a1
Jul 13 20:52:40.713520 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:40.713485    2584 generic.go:356] "Generic (PLEG): container finished" podID="8c816668-21a1-4fbd-8eb9-23f6c03317e7" containerID="48820afa4fd9fee4a0dc01c656187bef0c5e2611f27c76a05f958836a586aa67" exitCode=0
Jul 13 20:52:40.713875 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:40.713554    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2" event={"ID":"8c816668-21a1-4fbd-8eb9-23f6c03317e7","Type":"ContainerDied","Data":"48820afa4fd9fee4a0dc01c656187bef0c5e2611f27c76a05f958836a586aa67"}
Jul 13 20:52:40.713875 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:40.713592    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2" event={"ID":"8c816668-21a1-4fbd-8eb9-23f6c03317e7","Type":"ContainerStarted","Data":"a3fc17ecd052a1d455970e2daebb507f7c25b82e1a4937267e4a66a0600265a1"}
Jul 13 20:52:41.842361 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:41.842339    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:41.939996 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:41.939971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c816668-21a1-4fbd-8eb9-23f6c03317e7-maas-api-service-ca\") pod \"8c816668-21a1-4fbd-8eb9-23f6c03317e7\" (UID: \"8c816668-21a1-4fbd-8eb9-23f6c03317e7\") "
Jul 13 20:52:41.940300 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:41.940280    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c816668-21a1-4fbd-8eb9-23f6c03317e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c816668-21a1-4fbd-8eb9-23f6c03317e7" (UID: "8c816668-21a1-4fbd-8eb9-23f6c03317e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:52:42.040996 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:42.040931    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c816668-21a1-4fbd-8eb9-23f6c03317e7-maas-api-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:42.722051 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:42.722018    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2" event={"ID":"8c816668-21a1-4fbd-8eb9-23f6c03317e7","Type":"ContainerDied","Data":"a3fc17ecd052a1d455970e2daebb507f7c25b82e1a4937267e4a66a0600265a1"}
Jul 13 20:52:42.722051 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:42.722050    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-590516-fx6r2"
Jul 13 20:52:42.722278 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:42.722083    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3fc17ecd052a1d455970e2daebb507f7c25b82e1a4937267e4a66a0600265a1"
Jul 13 20:52:49.767535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:49.763099    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"]
Jul 13 20:52:49.767535 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:49.763329    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" podUID="df6db7a9-78d2-4603-bc99-2f6c3865d3da" containerName="maas-api" containerID="cri-o://f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a" gracePeriod=30
Jul 13 20:52:49.787092 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:49.787057    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:52:50.022122 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.022067    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:50.203937 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.203906    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6db7a9-78d2-4603-bc99-2f6c3865d3da-openshift-service-ca\") pod \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") "
Jul 13 20:52:50.204076 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.203979    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls\") pod \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") "
Jul 13 20:52:50.204076 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.204020    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhrjp\" (UniqueName: \"kubernetes.io/projected/df6db7a9-78d2-4603-bc99-2f6c3865d3da-kube-api-access-vhrjp\") pod \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\" (UID: \"df6db7a9-78d2-4603-bc99-2f6c3865d3da\") "
Jul 13 20:52:50.204243 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.204220    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6db7a9-78d2-4603-bc99-2f6c3865d3da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df6db7a9-78d2-4603-bc99-2f6c3865d3da" (UID: "df6db7a9-78d2-4603-bc99-2f6c3865d3da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:52:50.206027 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.205999    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6db7a9-78d2-4603-bc99-2f6c3865d3da-kube-api-access-vhrjp" (OuterVolumeSpecName: "kube-api-access-vhrjp") pod "df6db7a9-78d2-4603-bc99-2f6c3865d3da" (UID: "df6db7a9-78d2-4603-bc99-2f6c3865d3da"). InnerVolumeSpecName "kube-api-access-vhrjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:52:50.206131 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.206080    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df6db7a9-78d2-4603-bc99-2f6c3865d3da" (UID: "df6db7a9-78d2-4603-bc99-2f6c3865d3da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:52:50.304514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.304460    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6db7a9-78d2-4603-bc99-2f6c3865d3da-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:50.304514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.304480    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhrjp\" (UniqueName: \"kubernetes.io/projected/df6db7a9-78d2-4603-bc99-2f6c3865d3da-kube-api-access-vhrjp\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:50.304514 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.304489    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6db7a9-78d2-4603-bc99-2f6c3865d3da-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:52:50.753177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.753139    2584 generic.go:356] "Generic (PLEG): container finished" podID="df6db7a9-78d2-4603-bc99-2f6c3865d3da" containerID="f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a" exitCode=0
Jul 13 20:52:50.753310 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.753184    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"
Jul 13 20:52:50.753310 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.753189    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" event={"ID":"df6db7a9-78d2-4603-bc99-2f6c3865d3da","Type":"ContainerDied","Data":"f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a"}
Jul 13 20:52:50.753310 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.753241    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b" event={"ID":"df6db7a9-78d2-4603-bc99-2f6c3865d3da","Type":"ContainerDied","Data":"50a37510496686408c444857fac72a94f05922a352ce0eb171d4aa5bf1ea31c8"}
Jul 13 20:52:50.753310 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.753264    2584 scope.go:117] "RemoveContainer" containerID="f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a"
Jul 13 20:52:50.761410 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.761366    2584 scope.go:117] "RemoveContainer" containerID="f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a"
Jul 13 20:52:50.761662 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:52:50.761640    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a\": container with ID starting with f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a not found: ID does not exist" containerID="f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a"
Jul 13 20:52:50.761731 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.761673    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a"} err="failed to get container status \"f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a\": rpc error: code = NotFound desc = could not find container \"f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a\": container with ID starting with f3809048f9e09358b095f251d1c13964cd713401e1f19a3470a95acea118733a not found: ID does not exist"
Jul 13 20:52:50.768695 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.768669    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"]
Jul 13 20:52:50.770802 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:50.770783    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-590516-d54ddf8c8-tsq6b"]
Jul 13 20:52:52.637243 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:52.637201    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6db7a9-78d2-4603-bc99-2f6c3865d3da" path="/var/lib/kubelet/pods/df6db7a9-78d2-4603-bc99-2f6c3865d3da/volumes"
Jul 13 20:52:54.431843 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:52:54.431813    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:00.197799 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:00.197758    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:00.265863 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:00.265834    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:05.127229 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:05.127195    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:05.355992 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:05.355962    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:05.716707 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:05.716665    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:06.022369 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:06.022289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:06.399736 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:06.399701    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:09.174135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.174102    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:09.251358 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251330    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"]
Jul 13 20:53:09.251691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251673    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c816668-21a1-4fbd-8eb9-23f6c03317e7" containerName="revoke-keys"
Jul 13 20:53:09.251783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251694    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c816668-21a1-4fbd-8eb9-23f6c03317e7" containerName="revoke-keys"
Jul 13 20:53:09.251783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251708    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6db7a9-78d2-4603-bc99-2f6c3865d3da" containerName="maas-api"
Jul 13 20:53:09.251783 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251716    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6db7a9-78d2-4603-bc99-2f6c3865d3da" containerName="maas-api"
Jul 13 20:53:09.251929 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251783    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c816668-21a1-4fbd-8eb9-23f6c03317e7" containerName="revoke-keys"
Jul 13 20:53:09.251929 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.251797    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6db7a9-78d2-4603-bc99-2f6c3865d3da" containerName="maas-api"
Jul 13 20:53:09.254700 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.254678    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.257273 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.257252    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9swqx\""
Jul 13 20:53:09.257685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.257657    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 13 20:53:09.258462 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.258443    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 13 20:53:09.258581 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.258493    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-264ac7-serving-cert\""
Jul 13 20:53:09.264937 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.264914    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"]
Jul 13 20:53:09.288521 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.288496    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:09.327779 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.327746    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.327886 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.327796    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.327886 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.327844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqs6\" (UniqueName: \"kubernetes.io/projected/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-kube-api-access-qrqs6\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.428699 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.428637    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqs6\" (UniqueName: \"kubernetes.io/projected/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-kube-api-access-qrqs6\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.428699 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.428696    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.428857 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.428714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.428857 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:53:09.428801    2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-serving-cert: secret "maas-api-e2e-shared-a-264ac7-serving-cert" not found
Jul 13 20:53:09.428972 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:53:09.428863    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls podName:eb64262a-2c3b-4ab6-a764-ef6d9b88c26c nodeName:}" failed. No retries permitted until 2026-07-13 20:53:09.928846958 +0000 UTC m=+2323.829549603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls") pod "maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" (UID: "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c") : secret "maas-api-e2e-shared-a-264ac7-serving-cert" not found
Jul 13 20:53:09.429249 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.429231    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.442673 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.442649    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqs6\" (UniqueName: \"kubernetes.io/projected/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-kube-api-access-qrqs6\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.446623 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.446600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:09.545082 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.545040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:09.900849 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.900815    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:09.933073 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.933046    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.935516 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.935496    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls\") pod \"maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:09.999223 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:09.999193    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:10.168247 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:10.168164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:10.299706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:10.299680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"]
Jul 13 20:53:10.302349 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:53:10.302322    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb64262a_2c3b_4ab6_a764_ef6d9b88c26c.slice/crio-3f4cf81089f6c8cb118bd7b1f42683b523587e1800dc862ca1b5f53dea3f5f79 WatchSource:0}: Error finding container 3f4cf81089f6c8cb118bd7b1f42683b523587e1800dc862ca1b5f53dea3f5f79: Status 404 returned error can't find the container with id 3f4cf81089f6c8cb118bd7b1f42683b523587e1800dc862ca1b5f53dea3f5f79
Jul 13 20:53:10.836128 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:10.836093    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" event={"ID":"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c","Type":"ContainerStarted","Data":"3f4cf81089f6c8cb118bd7b1f42683b523587e1800dc862ca1b5f53dea3f5f79"}
Jul 13 20:53:11.842957 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:11.842922    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" event={"ID":"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c","Type":"ContainerStarted","Data":"1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969"}
Jul 13 20:53:11.843423 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:11.843037    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:11.859778 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:11.859734    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" podStartSLOduration=2.254419179 podStartE2EDuration="2.859719511s" totalImagesPullingTime="605.300332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:53:11.858250167 +0000 UTC m=+2325.758952829" watchObservedRunningTime="2026-07-13 20:53:11.859719511 +0000 UTC m=+2325.760422194"
Jul 13 20:53:13.615828 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:13.615781    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:13.996090 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:13.996013    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:14.246668 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:14.246589    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:14.586595 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:14.586546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:17.119612 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.119573    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:17.218414 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.218371    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:17.309106 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.309065    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"]
Jul 13 20:53:17.312340 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.312318    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.315210 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.315192    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-11843d-serving-cert\""
Jul 13 20:53:17.325211 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.325189    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"]
Jul 13 20:53:17.346163 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.346135    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:17.388304 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.388245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qhx\" (UniqueName: \"kubernetes.io/projected/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-kube-api-access-q4qhx\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.388304 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.388286    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-maas-api-tls\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.388463 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.388332    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-openshift-service-ca\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.489263 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.489240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-maas-api-tls\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.489367 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.489278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-openshift-service-ca\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.489367 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.489314    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qhx\" (UniqueName: \"kubernetes.io/projected/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-kube-api-access-q4qhx\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.489938 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.489920    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-openshift-service-ca\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.491858 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.491835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-maas-api-tls\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.497638 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.497615    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qhx\" (UniqueName: \"kubernetes.io/projected/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-kube-api-access-q4qhx\") pod \"maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.511290 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.511272    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:17.622651 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.622624    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:17.755016 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.754959    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"]
Jul 13 20:53:17.757379 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:53:17.757351    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2b9e5f_d09a_4b6a_be6a_2e5a67cee420.slice/crio-e37e0da736ac95fe628b70a258c4b22108df24e9dcdd36dd3758feedaa1a7122 WatchSource:0}: Error finding container e37e0da736ac95fe628b70a258c4b22108df24e9dcdd36dd3758feedaa1a7122: Status 404 returned error can't find the container with id e37e0da736ac95fe628b70a258c4b22108df24e9dcdd36dd3758feedaa1a7122
Jul 13 20:53:17.770509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.770478    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:17.853451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.853426    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:53:17.870685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.870619    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv" event={"ID":"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420","Type":"ContainerStarted","Data":"e37e0da736ac95fe628b70a258c4b22108df24e9dcdd36dd3758feedaa1a7122"}
Jul 13 20:53:17.872283 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:17.872252    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:18.876111 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:18.876076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv" event={"ID":"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420","Type":"ContainerStarted","Data":"8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071"}
Jul 13 20:53:18.876532 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:18.876188    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:18.893474 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:18.893437    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv" podStartSLOduration=1.445607316 podStartE2EDuration="1.893424362s" totalImagesPullingTime="447.817046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:53:18.892929733 +0000 UTC m=+2332.793632394" watchObservedRunningTime="2026-07-13 20:53:18.893424362 +0000 UTC m=+2332.794127023"
Jul 13 20:53:21.532982 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:21.532936    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:21.634730 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:21.634699    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:24.886028 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:24.886001    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 20:53:30.589298 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:30.589259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:30.683030 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:30.682992    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:35.183241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:35.183201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:35.295841 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:35.295799    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:38.492016 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:38.491984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:38.589668 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:38.589628    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:40.398594 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.398564    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"]
Jul 13 20:53:40.401518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.401503    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.405280 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.405254    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"kube-root-ca.crt\""
Jul 13 20:53:40.405435 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.405282    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"openshift-service-ca.crt\""
Jul 13 20:53:40.405435 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.405380    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"default-dockercfg-7lck8\""
Jul 13 20:53:40.405548 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.405528    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"auth-test-model-264ac7-kserve-self-signed-certs\""
Jul 13 20:53:40.409614 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.409596    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"]
Jul 13 20:53:40.415852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.415825    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:53:40.454170 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.454134    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrh8m\" (UniqueName: \"kubernetes.io/projected/ac51243a-de6e-4396-ad31-9180837d48e1-kube-api-access-hrh8m\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.454170 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.454176    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-tmp-dir\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.454340 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.454202    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-home\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.454340 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.454232    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac51243a-de6e-4396-ad31-9180837d48e1-tls-certs\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.454340 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.454325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-dshm\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.454488 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.454357    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-model-cache\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555549 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-dshm\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555654 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555559    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-model-cache\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555654 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555582    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrh8m\" (UniqueName: \"kubernetes.io/projected/ac51243a-de6e-4396-ad31-9180837d48e1-kube-api-access-hrh8m\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555758 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-tmp-dir\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555819 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555756    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-home\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555819 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555784    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac51243a-de6e-4396-ad31-9180837d48e1-tls-certs\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.555923 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.555870    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-model-cache\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.556050 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.556034    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-tmp-dir\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.556143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.556121    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-home\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.557879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.557860    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-dshm\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.558267 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.558250    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac51243a-de6e-4396-ad31-9180837d48e1-tls-certs\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.563219 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.563195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrh8m\" (UniqueName: \"kubernetes.io/projected/ac51243a-de6e-4396-ad31-9180837d48e1-kube-api-access-hrh8m\") pod \"auth-test-model-264ac7-kserve-79fbd45775-n4xr5\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") " pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.710481 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.710405    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:40.839628 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.839600    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"]
Jul 13 20:53:40.841838 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:53:40.841810    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac51243a_de6e_4396_ad31_9180837d48e1.slice/crio-6620e40ef74f9a856cd1238d1881e4a6e5c912cb792248d5576a0a184dc92c3f WatchSource:0}: Error finding container 6620e40ef74f9a856cd1238d1881e4a6e5c912cb792248d5576a0a184dc92c3f: Status 404 returned error can't find the container with id 6620e40ef74f9a856cd1238d1881e4a6e5c912cb792248d5576a0a184dc92c3f
Jul 13 20:53:40.964937 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.964857    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5" event={"ID":"ac51243a-de6e-4396-ad31-9180837d48e1","Type":"ContainerStarted","Data":"af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2"}
Jul 13 20:53:40.964937 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.964899    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5" event={"ID":"ac51243a-de6e-4396-ad31-9180837d48e1","Type":"ContainerStarted","Data":"6620e40ef74f9a856cd1238d1881e4a6e5c912cb792248d5576a0a184dc92c3f"}
Jul 13 20:53:40.981260 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:40.981215    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5" podStartSLOduration=0.981201764 podStartE2EDuration="981.201764ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:53:40.979570782 +0000 UTC m=+2354.880273444" watchObservedRunningTime="2026-07-13 20:53:40.981201764 +0000 UTC m=+2354.881904426"
Jul 13 20:53:50.711383 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:50.711348    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:50.711936 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:50.711409    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:50.723777 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:50.723749    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:53:51.021072 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:53:51.021000    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:54:11.295999 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.295961    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"]
Jul 13 20:54:11.299011 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.298993    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.302887 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.302859    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"openshift-service-ca.crt\""
Jul 13 20:54:11.303026 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.302903    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"auth-test-model-11843d-kserve-self-signed-certs\""
Jul 13 20:54:11.303026 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.302920    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"kube-root-ca.crt\""
Jul 13 20:54:11.303026 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.302906    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"default-dockercfg-zf4s9\""
Jul 13 20:54:11.307196 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.307176    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"]
Jul 13 20:54:11.377991 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.377964    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-home\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.378115 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.377999    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624vx\" (UniqueName: \"kubernetes.io/projected/804d8a83-a509-41ff-ae4b-3def81fc574c-kube-api-access-624vx\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.378115 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.378037    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-model-cache\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.378115 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.378084    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804d8a83-a509-41ff-ae4b-3def81fc574c-tls-certs\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.378115 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.378105    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-dshm\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.378247 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.378130    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-tmp-dir\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.399475 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.399449    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:54:11.478851 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.478822    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-home\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.478971 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.478861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624vx\" (UniqueName: \"kubernetes.io/projected/804d8a83-a509-41ff-ae4b-3def81fc574c-kube-api-access-624vx\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.478984    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-model-cache\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479036 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.479020    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804d8a83-a509-41ff-ae4b-3def81fc574c-tls-certs\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479148 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.479044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-dshm\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479148 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.479078    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-tmp-dir\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479288 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.479266    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-home\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479352 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.479295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-model-cache\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.479352 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.479327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-tmp-dir\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.481233 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.481215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-dshm\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.481639 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.481619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804d8a83-a509-41ff-ae4b-3def81fc574c-tls-certs\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.487236 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.487213    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-624vx\" (UniqueName: \"kubernetes.io/projected/804d8a83-a509-41ff-ae4b-3def81fc574c-kube-api-access-624vx\") pod \"auth-test-model-11843d-kserve-7ddd75fff9-64hwd\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") " pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.609016 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.608991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:11.735220 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:11.735195    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"]
Jul 13 20:54:11.737786 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:54:11.737752    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod804d8a83_a509_41ff_ae4b_3def81fc574c.slice/crio-0071dba74cf9d72f30f969db1b1066c85885168344a013d7200f247c1f7db9fb WatchSource:0}: Error finding container 0071dba74cf9d72f30f969db1b1066c85885168344a013d7200f247c1f7db9fb: Status 404 returned error can't find the container with id 0071dba74cf9d72f30f969db1b1066c85885168344a013d7200f247c1f7db9fb
Jul 13 20:54:12.098125 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:12.098086    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd" event={"ID":"804d8a83-a509-41ff-ae4b-3def81fc574c","Type":"ContainerStarted","Data":"88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf"}
Jul 13 20:54:12.098321 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:12.098132    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd" event={"ID":"804d8a83-a509-41ff-ae4b-3def81fc574c","Type":"ContainerStarted","Data":"0071dba74cf9d72f30f969db1b1066c85885168344a013d7200f247c1f7db9fb"}
Jul 13 20:54:12.115656 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:12.115603    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd" podStartSLOduration=1.115585177 podStartE2EDuration="1.115585177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:54:12.113430645 +0000 UTC m=+2386.014133327" watchObservedRunningTime="2026-07-13 20:54:12.115585177 +0000 UTC m=+2386.016287840"
Jul 13 20:54:21.609289 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:21.609252    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:21.609289 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:21.609297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:21.621869 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:21.621847    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:22.150273 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:22.150248    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:54:50.363298 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:54:50.363174    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:55:10.435021 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:55:10.434986    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:55:45.477946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:55:45.477904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:12.994486 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:12.994445    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:13.094548 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:13.094521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:17.173383 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:17.173354    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"]
Jul 13 20:56:17.199648 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:17.199621    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:18.170170 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:18.170139    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"]
Jul 13 20:56:18.187427 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:18.187379    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:24.783639 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:24.783606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:29.612086 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:29.612042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:56:32.193447 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.193384    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5" podUID="ac51243a-de6e-4396-ad31-9180837d48e1" containerName="main" containerID="cri-o://af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2" gracePeriod=45
Jul 13 20:56:32.434211 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.434182    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:56:32.553357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553270    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-model-cache\") pod \"ac51243a-de6e-4396-ad31-9180837d48e1\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") "
Jul 13 20:56:32.553357 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553328    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-home\") pod \"ac51243a-de6e-4396-ad31-9180837d48e1\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") "
Jul 13 20:56:32.553619 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553366    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-tmp-dir\") pod \"ac51243a-de6e-4396-ad31-9180837d48e1\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") "
Jul 13 20:56:32.553619 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553432    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrh8m\" (UniqueName: \"kubernetes.io/projected/ac51243a-de6e-4396-ad31-9180837d48e1-kube-api-access-hrh8m\") pod \"ac51243a-de6e-4396-ad31-9180837d48e1\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") "
Jul 13 20:56:32.553619 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553469    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-dshm\") pod \"ac51243a-de6e-4396-ad31-9180837d48e1\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") "
Jul 13 20:56:32.553619 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553493    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac51243a-de6e-4396-ad31-9180837d48e1-tls-certs\") pod \"ac51243a-de6e-4396-ad31-9180837d48e1\" (UID: \"ac51243a-de6e-4396-ad31-9180837d48e1\") "
Jul 13 20:56:32.553619 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553562    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac51243a-de6e-4396-ad31-9180837d48e1" (UID: "ac51243a-de6e-4396-ad31-9180837d48e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:32.553876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553625    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-home" (OuterVolumeSpecName: "home") pod "ac51243a-de6e-4396-ad31-9180837d48e1" (UID: "ac51243a-de6e-4396-ad31-9180837d48e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:32.553876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553671    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac51243a-de6e-4396-ad31-9180837d48e1" (UID: "ac51243a-de6e-4396-ad31-9180837d48e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:32.553876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553798    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:32.553876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553815    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:32.553876 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.553856    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:32.555895 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.555870    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-dshm" (OuterVolumeSpecName: "dshm") pod "ac51243a-de6e-4396-ad31-9180837d48e1" (UID: "ac51243a-de6e-4396-ad31-9180837d48e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:32.556017 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.555899    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac51243a-de6e-4396-ad31-9180837d48e1-kube-api-access-hrh8m" (OuterVolumeSpecName: "kube-api-access-hrh8m") pod "ac51243a-de6e-4396-ad31-9180837d48e1" (UID: "ac51243a-de6e-4396-ad31-9180837d48e1"). InnerVolumeSpecName "kube-api-access-hrh8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:56:32.556079 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.556040    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac51243a-de6e-4396-ad31-9180837d48e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac51243a-de6e-4396-ad31-9180837d48e1" (UID: "ac51243a-de6e-4396-ad31-9180837d48e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:56:32.654167 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.654143    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrh8m\" (UniqueName: \"kubernetes.io/projected/ac51243a-de6e-4396-ad31-9180837d48e1-kube-api-access-hrh8m\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:32.654167 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.654165    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac51243a-de6e-4396-ad31-9180837d48e1-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:32.654315 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.654174    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac51243a-de6e-4396-ad31-9180837d48e1-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:32.702718 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.702685    2584 generic.go:356] "Generic (PLEG): container finished" podID="ac51243a-de6e-4396-ad31-9180837d48e1" containerID="af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2" exitCode=0
Jul 13 20:56:32.702868 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.702723    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5" event={"ID":"ac51243a-de6e-4396-ad31-9180837d48e1","Type":"ContainerDied","Data":"af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2"}
Jul 13 20:56:32.702868 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.702743    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"
Jul 13 20:56:32.702868 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.702756    2584 scope.go:117] "RemoveContainer" containerID="af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2"
Jul 13 20:56:32.702868 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.702747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5" event={"ID":"ac51243a-de6e-4396-ad31-9180837d48e1","Type":"ContainerDied","Data":"6620e40ef74f9a856cd1238d1881e4a6e5c912cb792248d5576a0a184dc92c3f"}
Jul 13 20:56:32.710832 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.710815    2584 scope.go:117] "RemoveContainer" containerID="af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2"
Jul 13 20:56:32.711124 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:56:32.711102    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2\": container with ID starting with af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2 not found: ID does not exist" containerID="af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2"
Jul 13 20:56:32.711204 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.711133    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2"} err="failed to get container status \"af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2\": rpc error: code = NotFound desc = could not find container \"af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2\": container with ID starting with af3dc483fb58b8d8f32c9a9439794ae25f4a415dd5bb992dca6ab76a9fa762e2 not found: ID does not exist"
Jul 13 20:56:32.718853 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.718830    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"]
Jul 13 20:56:32.723657 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:32.723635    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/auth-test-model-264ac7-kserve-79fbd45775-n4xr5"]
Jul 13 20:56:33.191311 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.191264    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd" podUID="804d8a83-a509-41ff-ae4b-3def81fc574c" containerName="main" containerID="cri-o://88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf" gracePeriod=45
Jul 13 20:56:33.458112 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.458089    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:56:33.561950 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.561918    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-model-cache\") pod \"804d8a83-a509-41ff-ae4b-3def81fc574c\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") "
Jul 13 20:56:33.562135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.561991    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804d8a83-a509-41ff-ae4b-3def81fc574c-tls-certs\") pod \"804d8a83-a509-41ff-ae4b-3def81fc574c\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") "
Jul 13 20:56:33.562135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562015    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-dshm\") pod \"804d8a83-a509-41ff-ae4b-3def81fc574c\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") "
Jul 13 20:56:33.562135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562041    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-home\") pod \"804d8a83-a509-41ff-ae4b-3def81fc574c\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") "
Jul 13 20:56:33.562135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562063    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-tmp-dir\") pod \"804d8a83-a509-41ff-ae4b-3def81fc574c\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") "
Jul 13 20:56:33.562135 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562089    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-624vx\" (UniqueName: \"kubernetes.io/projected/804d8a83-a509-41ff-ae4b-3def81fc574c-kube-api-access-624vx\") pod \"804d8a83-a509-41ff-ae4b-3def81fc574c\" (UID: \"804d8a83-a509-41ff-ae4b-3def81fc574c\") "
Jul 13 20:56:33.562379 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562148    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-model-cache" (OuterVolumeSpecName: "model-cache") pod "804d8a83-a509-41ff-ae4b-3def81fc574c" (UID: "804d8a83-a509-41ff-ae4b-3def81fc574c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:33.562379 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562254    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "804d8a83-a509-41ff-ae4b-3def81fc574c" (UID: "804d8a83-a509-41ff-ae4b-3def81fc574c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:33.562379 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562310    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-home" (OuterVolumeSpecName: "home") pod "804d8a83-a509-41ff-ae4b-3def81fc574c" (UID: "804d8a83-a509-41ff-ae4b-3def81fc574c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:33.562570 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562427    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:33.562570 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562442    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:33.562570 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.562451    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:33.564327 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.564294    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-dshm" (OuterVolumeSpecName: "dshm") pod "804d8a83-a509-41ff-ae4b-3def81fc574c" (UID: "804d8a83-a509-41ff-ae4b-3def81fc574c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:56:33.564327 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.564307    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804d8a83-a509-41ff-ae4b-3def81fc574c-kube-api-access-624vx" (OuterVolumeSpecName: "kube-api-access-624vx") pod "804d8a83-a509-41ff-ae4b-3def81fc574c" (UID: "804d8a83-a509-41ff-ae4b-3def81fc574c"). InnerVolumeSpecName "kube-api-access-624vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:56:33.564518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.564424    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804d8a83-a509-41ff-ae4b-3def81fc574c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "804d8a83-a509-41ff-ae4b-3def81fc574c" (UID: "804d8a83-a509-41ff-ae4b-3def81fc574c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:56:33.663177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.663148    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/804d8a83-a509-41ff-ae4b-3def81fc574c-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:33.663177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.663170    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/804d8a83-a509-41ff-ae4b-3def81fc574c-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:33.663177 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.663180    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624vx\" (UniqueName: \"kubernetes.io/projected/804d8a83-a509-41ff-ae4b-3def81fc574c-kube-api-access-624vx\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:56:33.716454 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.716332    2584 generic.go:356] "Generic (PLEG): container finished" podID="804d8a83-a509-41ff-ae4b-3def81fc574c" containerID="88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf" exitCode=0
Jul 13 20:56:33.716609 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.716483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd" event={"ID":"804d8a83-a509-41ff-ae4b-3def81fc574c","Type":"ContainerDied","Data":"88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf"}
Jul 13 20:56:33.716609 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.716536    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd" event={"ID":"804d8a83-a509-41ff-ae4b-3def81fc574c","Type":"ContainerDied","Data":"0071dba74cf9d72f30f969db1b1066c85885168344a013d7200f247c1f7db9fb"}
Jul 13 20:56:33.716609 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.716558    2584 scope.go:117] "RemoveContainer" containerID="88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf"
Jul 13 20:56:33.716776 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.716666    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"
Jul 13 20:56:33.743934 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.743911    2584 scope.go:117] "RemoveContainer" containerID="88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf"
Jul 13 20:56:33.744240 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:56:33.744221    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf\": container with ID starting with 88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf not found: ID does not exist" containerID="88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf"
Jul 13 20:56:33.744305 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.744252    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf"} err="failed to get container status \"88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf\": rpc error: code = NotFound desc = could not find container \"88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf\": container with ID starting with 88f93dffd13bce0cb673539ccd30c4f8bda1627b5fd2256b5c8e6732df5a0aaf not found: ID does not exist"
Jul 13 20:56:33.752190 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.752163    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"]
Jul 13 20:56:33.754509 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:33.754487    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/auth-test-model-11843d-kserve-7ddd75fff9-64hwd"]
Jul 13 20:56:34.637260 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:34.637213    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="804d8a83-a509-41ff-ae4b-3def81fc574c" path="/var/lib/kubelet/pods/804d8a83-a509-41ff-ae4b-3def81fc574c/volumes"
Jul 13 20:56:34.637777 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:56:34.637747    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac51243a-de6e-4396-ad31-9180837d48e1" path="/var/lib/kubelet/pods/ac51243a-de6e-4396-ad31-9180837d48e1/volumes"
Jul 13 20:57:06.396718 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.396684    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"]
Jul 13 20:57:06.397184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.396946    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac51243a-de6e-4396-ad31-9180837d48e1" containerName="main"
Jul 13 20:57:06.397184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.396957    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac51243a-de6e-4396-ad31-9180837d48e1" containerName="main"
Jul 13 20:57:06.397184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.396980    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804d8a83-a509-41ff-ae4b-3def81fc574c" containerName="main"
Jul 13 20:57:06.397184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.396987    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="804d8a83-a509-41ff-ae4b-3def81fc574c" containerName="main"
Jul 13 20:57:06.397184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.397030    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac51243a-de6e-4396-ad31-9180837d48e1" containerName="main"
Jul 13 20:57:06.397184 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.397040    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="804d8a83-a509-41ff-ae4b-3def81fc574c" containerName="main"
Jul 13 20:57:06.399714 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.399698    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.403484 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.403461    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"kube-root-ca.crt\""
Jul 13 20:57:06.403484 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.403461    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"openshift-service-ca.crt\""
Jul 13 20:57:06.403685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.403491    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"sub-test-model-11843d-kserve-self-signed-certs\""
Jul 13 20:57:06.403685 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.403498    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-11843d\"/\"default-dockercfg-zf4s9\""
Jul 13 20:57:06.410846 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.410823    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"]
Jul 13 20:57:06.489008 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.488981    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:57:06.499153 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.499126    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-dshm\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.499266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.499159    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-tmp-dir\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.499266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.499185    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77ce34-8577-484e-b796-fae6feb3da2d-tls-certs\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.499266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.499259    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqf6\" (UniqueName: \"kubernetes.io/projected/fd77ce34-8577-484e-b796-fae6feb3da2d-kube-api-access-clqf6\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.499434 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.499297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-home\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.499434 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.499325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-model-cache\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.599935 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.599898    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-model-cache\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600064 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-dshm\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-tmp-dir\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600133 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77ce34-8577-484e-b796-fae6feb3da2d-tls-certs\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqf6\" (UniqueName: \"kubernetes.io/projected/fd77ce34-8577-484e-b796-fae6feb3da2d-kube-api-access-clqf6\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600186    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-model-cache\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600241 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600185    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-home\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600505 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600478    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-tmp-dir\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.600576 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.600523    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-home\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.602320 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.602303    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-dshm\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.602653 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.602635    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77ce34-8577-484e-b796-fae6feb3da2d-tls-certs\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.609047 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.609026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqf6\" (UniqueName: \"kubernetes.io/projected/fd77ce34-8577-484e-b796-fae6feb3da2d-kube-api-access-clqf6\") pod \"sub-test-model-11843d-kserve-7674758df7-h2vrm\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") " pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.708085 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.708027    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:06.837949 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.837920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"]
Jul 13 20:57:06.841329 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:57:06.841302    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd77ce34_8577_484e_b796_fae6feb3da2d.slice/crio-53a5be6c21a025f555ecfb4a558c49d0e9f657347a4ac48a4d9dad93d14327d1 WatchSource:0}: Error finding container 53a5be6c21a025f555ecfb4a558c49d0e9f657347a4ac48a4d9dad93d14327d1: Status 404 returned error can't find the container with id 53a5be6c21a025f555ecfb4a558c49d0e9f657347a4ac48a4d9dad93d14327d1
Jul 13 20:57:06.869066 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:06.869040    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm" event={"ID":"fd77ce34-8577-484e-b796-fae6feb3da2d","Type":"ContainerStarted","Data":"53a5be6c21a025f555ecfb4a558c49d0e9f657347a4ac48a4d9dad93d14327d1"}
Jul 13 20:57:07.875486 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:07.875439    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm" event={"ID":"fd77ce34-8577-484e-b796-fae6feb3da2d","Type":"ContainerStarted","Data":"45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24"}
Jul 13 20:57:07.892092 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:07.892040    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm" podStartSLOduration=1.892025142 podStartE2EDuration="1.892025142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:57:07.890568258 +0000 UTC m=+2561.791270921" watchObservedRunningTime="2026-07-13 20:57:07.892025142 +0000 UTC m=+2561.792727804"
Jul 13 20:57:16.708354 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:16.708313    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:16.708354 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:16.708358    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:16.720517 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:16.720496    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:16.925078 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:16.925051    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:57:52.640510 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:52.640424    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:57:53.481020 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:53.480989    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"]
Jul 13 20:57:53.741114 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:53.741023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:57:56.992966 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:56.992926    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"]
Jul 13 20:57:56.996380 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:56.996362    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.000448 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.000421    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"default-dockercfg-7lck8\""
Jul 13 20:57:57.000578 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.000427    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"kube-root-ca.crt\""
Jul 13 20:57:57.000578 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.000464    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"openshift-service-ca.crt\""
Jul 13 20:57:57.000578 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.000428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-264ac7\"/\"rate-test-model-264ac7-kserve-self-signed-certs\""
Jul 13 20:57:57.005818 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.005793    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"]
Jul 13 20:57:57.028921 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.028885    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:57:57.083823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.083798    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-home\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.083939 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.083871    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-dshm\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.083939 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.083890    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-model-cache\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.083939 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.083905    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srnqc\" (UniqueName: \"kubernetes.io/projected/5c314f52-768e-40bc-bf6c-7fb227b14f90-kube-api-access-srnqc\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.084110 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.083962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c314f52-768e-40bc-bf6c-7fb227b14f90-tls-certs\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.084110 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.083985    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-tmp-dir\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185137 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185111    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-dshm\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185142    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-model-cache\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185158    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srnqc\" (UniqueName: \"kubernetes.io/projected/5c314f52-768e-40bc-bf6c-7fb227b14f90-kube-api-access-srnqc\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c314f52-768e-40bc-bf6c-7fb227b14f90-tls-certs\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185211    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-tmp-dir\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185277 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185240    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-home\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185654    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-model-cache\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185691 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185681    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-home\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.185864 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.185698    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-tmp-dir\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.187692 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.187664    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-dshm\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.187813 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.187798    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c314f52-768e-40bc-bf6c-7fb227b14f90-tls-certs\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.194450 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.194430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srnqc\" (UniqueName: \"kubernetes.io/projected/5c314f52-768e-40bc-bf6c-7fb227b14f90-kube-api-access-srnqc\") pod \"rate-test-model-264ac7-kserve-685c5cdd7-lc4zr\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") " pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.306187 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.306127    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:57:57.436898 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:57.436862    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"]
Jul 13 20:57:57.440110 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:57:57.440086    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c314f52_768e_40bc_bf6c_7fb227b14f90.slice/crio-de633bbce80c4c10bcd2dde28f59dc53333bf0a558bacd850d30f1356786a957 WatchSource:0}: Error finding container de633bbce80c4c10bcd2dde28f59dc53333bf0a558bacd850d30f1356786a957: Status 404 returned error can't find the container with id de633bbce80c4c10bcd2dde28f59dc53333bf0a558bacd850d30f1356786a957
Jul 13 20:57:58.090044 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:58.090007    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr" event={"ID":"5c314f52-768e-40bc-bf6c-7fb227b14f90","Type":"ContainerStarted","Data":"fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb"}
Jul 13 20:57:58.090466 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:58.090051    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr" event={"ID":"5c314f52-768e-40bc-bf6c-7fb227b14f90","Type":"ContainerStarted","Data":"de633bbce80c4c10bcd2dde28f59dc53333bf0a558bacd850d30f1356786a957"}
Jul 13 20:57:58.107377 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:57:58.107312    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr" podStartSLOduration=2.107298474 podStartE2EDuration="2.107298474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:57:58.105727794 +0000 UTC m=+2612.006430455" watchObservedRunningTime="2026-07-13 20:57:58.107298474 +0000 UTC m=+2612.008001135"
Jul 13 20:58:07.306702 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:07.306666    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:58:07.307127 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:07.306775    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:58:07.319636 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:07.319611    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:58:08.142183 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.142156    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:58:08.501488 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.501375    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm" podUID="fd77ce34-8577-484e-b796-fae6feb3da2d" containerName="main" containerID="cri-o://45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24" gracePeriod=45
Jul 13 20:58:08.746441 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.746414    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:58:08.881461 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881434    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clqf6\" (UniqueName: \"kubernetes.io/projected/fd77ce34-8577-484e-b796-fae6feb3da2d-kube-api-access-clqf6\") pod \"fd77ce34-8577-484e-b796-fae6feb3da2d\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") "
Jul 13 20:58:08.881603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881470    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-tmp-dir\") pod \"fd77ce34-8577-484e-b796-fae6feb3da2d\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") "
Jul 13 20:58:08.881603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881493    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77ce34-8577-484e-b796-fae6feb3da2d-tls-certs\") pod \"fd77ce34-8577-484e-b796-fae6feb3da2d\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") "
Jul 13 20:58:08.881603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881510    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-dshm\") pod \"fd77ce34-8577-484e-b796-fae6feb3da2d\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") "
Jul 13 20:58:08.881603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881532    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-model-cache\") pod \"fd77ce34-8577-484e-b796-fae6feb3da2d\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") "
Jul 13 20:58:08.881819 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881634    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-home\") pod \"fd77ce34-8577-484e-b796-fae6feb3da2d\" (UID: \"fd77ce34-8577-484e-b796-fae6feb3da2d\") "
Jul 13 20:58:08.881819 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881737    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd77ce34-8577-484e-b796-fae6feb3da2d" (UID: "fd77ce34-8577-484e-b796-fae6feb3da2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:58:08.881917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881821    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd77ce34-8577-484e-b796-fae6feb3da2d" (UID: "fd77ce34-8577-484e-b796-fae6feb3da2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:58:08.881917 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881846    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-home" (OuterVolumeSpecName: "home") pod "fd77ce34-8577-484e-b796-fae6feb3da2d" (UID: "fd77ce34-8577-484e-b796-fae6feb3da2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:58:08.881990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881921    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:58:08.881990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881935    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:58:08.881990 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.881944    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:58:08.883635 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.883613    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-dshm" (OuterVolumeSpecName: "dshm") pod "fd77ce34-8577-484e-b796-fae6feb3da2d" (UID: "fd77ce34-8577-484e-b796-fae6feb3da2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:58:08.884120 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.884102    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd77ce34-8577-484e-b796-fae6feb3da2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd77ce34-8577-484e-b796-fae6feb3da2d" (UID: "fd77ce34-8577-484e-b796-fae6feb3da2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:58:08.884182 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.884130    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd77ce34-8577-484e-b796-fae6feb3da2d-kube-api-access-clqf6" (OuterVolumeSpecName: "kube-api-access-clqf6") pod "fd77ce34-8577-484e-b796-fae6feb3da2d" (UID: "fd77ce34-8577-484e-b796-fae6feb3da2d"). InnerVolumeSpecName "kube-api-access-clqf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:58:08.982451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.982418    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqf6\" (UniqueName: \"kubernetes.io/projected/fd77ce34-8577-484e-b796-fae6feb3da2d-kube-api-access-clqf6\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:58:08.982451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.982449    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd77ce34-8577-484e-b796-fae6feb3da2d-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:58:08.982451 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:08.982460    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd77ce34-8577-484e-b796-fae6feb3da2d-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:58:09.136492 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.136418    2584 generic.go:356] "Generic (PLEG): container finished" podID="fd77ce34-8577-484e-b796-fae6feb3da2d" containerID="45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24" exitCode=0
Jul 13 20:58:09.136492 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.136456    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"
Jul 13 20:58:09.136669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.136496    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm" event={"ID":"fd77ce34-8577-484e-b796-fae6feb3da2d","Type":"ContainerDied","Data":"45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24"}
Jul 13 20:58:09.136669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.136543    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm" event={"ID":"fd77ce34-8577-484e-b796-fae6feb3da2d","Type":"ContainerDied","Data":"53a5be6c21a025f555ecfb4a558c49d0e9f657347a4ac48a4d9dad93d14327d1"}
Jul 13 20:58:09.136669 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.136565    2584 scope.go:117] "RemoveContainer" containerID="45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24"
Jul 13 20:58:09.145175 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.145158    2584 scope.go:117] "RemoveContainer" containerID="45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24"
Jul 13 20:58:09.145433 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:58:09.145417    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24\": container with ID starting with 45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24 not found: ID does not exist" containerID="45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24"
Jul 13 20:58:09.145502 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.145440    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24"} err="failed to get container status \"45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24\": rpc error: code = NotFound desc = could not find container \"45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24\": container with ID starting with 45fd32053ae1a6f36600dd26c2981bd163f7e70c94fc98891eb7287fefdb6b24 not found: ID does not exist"
Jul 13 20:58:09.157445 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.157417    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"]
Jul 13 20:58:09.160139 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:09.160120    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-11843d/sub-test-model-11843d-kserve-7674758df7-h2vrm"]
Jul 13 20:58:10.636959 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:10.636926    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd77ce34-8577-484e-b796-fae6feb3da2d" path="/var/lib/kubelet/pods/fd77ce34-8577-484e-b796-fae6feb3da2d/volumes"
Jul 13 20:58:25.335679 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:25.335638    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:58:31.409441 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:31.409382    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:58:31.510109 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:31.510072    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:58:36.058914 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:36.058871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:58:36.159326 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:36.159290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:58:39.297862 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:39.297816    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:58:39.399957 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:58:39.399926    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:07.524409 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:07.524359    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"]
Jul 13 20:59:07.544448 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:07.544419    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:08.412820 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:08.412779    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:12.160793 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.160753    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7326-140510219653680"]
Jul 13 20:59:12.161173 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.161044    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd77ce34-8577-484e-b796-fae6feb3da2d" containerName="main"
Jul 13 20:59:12.161173 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.161056    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd77ce34-8577-484e-b796-fae6feb3da2d" containerName="main"
Jul 13 20:59:12.161173 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.161123    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd77ce34-8577-484e-b796-fae6feb3da2d" containerName="main"
Jul 13 20:59:12.164001 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.163983    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:12.166455 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.166428    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-kxvn6\""
Jul 13 20:59:12.169527 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.169500    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7326-140510219653680"]
Jul 13 20:59:12.244121 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.244096    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fx6d\" (UniqueName: \"kubernetes.io/projected/575eb5ca-0a0c-4c09-b65e-c0fed0855db8-kube-api-access-9fx6d\") pod \"test-curl-7326-140510219653680\" (UID: \"575eb5ca-0a0c-4c09-b65e-c0fed0855db8\") " pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:12.345345 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.345320    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fx6d\" (UniqueName: \"kubernetes.io/projected/575eb5ca-0a0c-4c09-b65e-c0fed0855db8-kube-api-access-9fx6d\") pod \"test-curl-7326-140510219653680\" (UID: \"575eb5ca-0a0c-4c09-b65e-c0fed0855db8\") " pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:12.353695 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.353672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fx6d\" (UniqueName: \"kubernetes.io/projected/575eb5ca-0a0c-4c09-b65e-c0fed0855db8-kube-api-access-9fx6d\") pod \"test-curl-7326-140510219653680\" (UID: \"575eb5ca-0a0c-4c09-b65e-c0fed0855db8\") " pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:12.474030 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.473966    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:12.605932 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.605908    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7326-140510219653680"]
Jul 13 20:59:12.607361 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:59:12.607333    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod575eb5ca_0a0c_4c09_b65e_c0fed0855db8.slice/crio-7c7f270d54a06fe9d5cb8bda4aae87e0112ec76e5ed9e893cc5d6de2fd1c8006 WatchSource:0}: Error finding container 7c7f270d54a06fe9d5cb8bda4aae87e0112ec76e5ed9e893cc5d6de2fd1c8006: Status 404 returned error can't find the container with id 7c7f270d54a06fe9d5cb8bda4aae87e0112ec76e5ed9e893cc5d6de2fd1c8006
Jul 13 20:59:12.608804 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:12.608783    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:59:13.422925 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:13.422897    2584 generic.go:356] "Generic (PLEG): container finished" podID="575eb5ca-0a0c-4c09-b65e-c0fed0855db8" containerID="c65ddb1402b2aa702b6062e1d192da223c79cb4ba66e9c035a77e4e1510b06a8" exitCode=0
Jul 13 20:59:13.423245 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:13.422941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7326-140510219653680" event={"ID":"575eb5ca-0a0c-4c09-b65e-c0fed0855db8","Type":"ContainerDied","Data":"c65ddb1402b2aa702b6062e1d192da223c79cb4ba66e9c035a77e4e1510b06a8"}
Jul 13 20:59:13.423245 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:13.422990    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7326-140510219653680" event={"ID":"575eb5ca-0a0c-4c09-b65e-c0fed0855db8","Type":"ContainerStarted","Data":"7c7f270d54a06fe9d5cb8bda4aae87e0112ec76e5ed9e893cc5d6de2fd1c8006"}
Jul 13 20:59:14.559172 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.559151    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:14.577795 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.577776    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7326-140510219653680_575eb5ca-0a0c-4c09-b65e-c0fed0855db8/test-curl-7326-140510219653680/0.log"
Jul 13 20:59:14.599932 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.599906    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7326-140510219653680"]
Jul 13 20:59:14.603471 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.603451    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7326-140510219653680"]
Jul 13 20:59:14.665460 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.665440    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fx6d\" (UniqueName: \"kubernetes.io/projected/575eb5ca-0a0c-4c09-b65e-c0fed0855db8-kube-api-access-9fx6d\") pod \"575eb5ca-0a0c-4c09-b65e-c0fed0855db8\" (UID: \"575eb5ca-0a0c-4c09-b65e-c0fed0855db8\") "
Jul 13 20:59:14.667527 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.667503    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575eb5ca-0a0c-4c09-b65e-c0fed0855db8-kube-api-access-9fx6d" (OuterVolumeSpecName: "kube-api-access-9fx6d") pod "575eb5ca-0a0c-4c09-b65e-c0fed0855db8" (UID: "575eb5ca-0a0c-4c09-b65e-c0fed0855db8"). InnerVolumeSpecName "kube-api-access-9fx6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:59:14.766738 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:14.766680    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fx6d\" (UniqueName: \"kubernetes.io/projected/575eb5ca-0a0c-4c09-b65e-c0fed0855db8-kube-api-access-9fx6d\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:15.439083 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:15.439053    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510219653680"
Jul 13 20:59:15.439278 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:15.439086    2584 scope.go:117] "RemoveContainer" containerID="c65ddb1402b2aa702b6062e1d192da223c79cb4ba66e9c035a77e4e1510b06a8"
Jul 13 20:59:16.637714 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:16.637599    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575eb5ca-0a0c-4c09-b65e-c0fed0855db8" path="/var/lib/kubelet/pods/575eb5ca-0a0c-4c09-b65e-c0fed0855db8/volumes"
Jul 13 20:59:17.172508 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.172470    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7326-140510225509232"]
Jul 13 20:59:17.172837 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.172819    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575eb5ca-0a0c-4c09-b65e-c0fed0855db8" containerName="test-curl-7326-140510219653680"
Jul 13 20:59:17.172924 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.172840    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="575eb5ca-0a0c-4c09-b65e-c0fed0855db8" containerName="test-curl-7326-140510219653680"
Jul 13 20:59:17.172981 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.172924    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="575eb5ca-0a0c-4c09-b65e-c0fed0855db8" containerName="test-curl-7326-140510219653680"
Jul 13 20:59:17.177225 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.177204    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:17.179879 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.179861    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-kxvn6\""
Jul 13 20:59:17.182741 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.182698    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7326-140510225509232"]
Jul 13 20:59:17.283662 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.283634    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhcj\" (UniqueName: \"kubernetes.io/projected/13e5ea94-79a2-4799-a1a8-ece9b70c63f3-kube-api-access-5nhcj\") pod \"test-curl-7326-140510225509232\" (UID: \"13e5ea94-79a2-4799-a1a8-ece9b70c63f3\") " pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:17.385011 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.384982    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhcj\" (UniqueName: \"kubernetes.io/projected/13e5ea94-79a2-4799-a1a8-ece9b70c63f3-kube-api-access-5nhcj\") pod \"test-curl-7326-140510225509232\" (UID: \"13e5ea94-79a2-4799-a1a8-ece9b70c63f3\") " pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:17.393345 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.393319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhcj\" (UniqueName: \"kubernetes.io/projected/13e5ea94-79a2-4799-a1a8-ece9b70c63f3-kube-api-access-5nhcj\") pod \"test-curl-7326-140510225509232\" (UID: \"13e5ea94-79a2-4799-a1a8-ece9b70c63f3\") " pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:17.486524 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.486469    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:17.614331 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:17.614307    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7326-140510225509232"]
Jul 13 20:59:17.616791 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:59:17.616758    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13e5ea94_79a2_4799_a1a8_ece9b70c63f3.slice/crio-e96d71c1f8e7b2c8bfa1416ecfc50bba32046bfbe3db961a759105b73a3edfda WatchSource:0}: Error finding container e96d71c1f8e7b2c8bfa1416ecfc50bba32046bfbe3db961a759105b73a3edfda: Status 404 returned error can't find the container with id e96d71c1f8e7b2c8bfa1416ecfc50bba32046bfbe3db961a759105b73a3edfda
Jul 13 20:59:18.453915 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:18.453882    2584 generic.go:356] "Generic (PLEG): container finished" podID="13e5ea94-79a2-4799-a1a8-ece9b70c63f3" containerID="3d9e71f6663c032ac6b01096de9600cce8bf47471dc3d445afffbc79fa3220a2" exitCode=0
Jul 13 20:59:18.454244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:18.453966    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7326-140510225509232" event={"ID":"13e5ea94-79a2-4799-a1a8-ece9b70c63f3","Type":"ContainerDied","Data":"3d9e71f6663c032ac6b01096de9600cce8bf47471dc3d445afffbc79fa3220a2"}
Jul 13 20:59:18.454244 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:18.454005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7326-140510225509232" event={"ID":"13e5ea94-79a2-4799-a1a8-ece9b70c63f3","Type":"ContainerStarted","Data":"e96d71c1f8e7b2c8bfa1416ecfc50bba32046bfbe3db961a759105b73a3edfda"}
Jul 13 20:59:19.590075 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.590052    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:19.609980 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.609955    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7326-140510225509232_13e5ea94-79a2-4799-a1a8-ece9b70c63f3/test-curl-7326-140510225509232/0.log"
Jul 13 20:59:19.631833 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.631811    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7326-140510225509232"]
Jul 13 20:59:19.634093 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.634071    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7326-140510225509232"]
Jul 13 20:59:19.703716 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.703691    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nhcj\" (UniqueName: \"kubernetes.io/projected/13e5ea94-79a2-4799-a1a8-ece9b70c63f3-kube-api-access-5nhcj\") pod \"13e5ea94-79a2-4799-a1a8-ece9b70c63f3\" (UID: \"13e5ea94-79a2-4799-a1a8-ece9b70c63f3\") "
Jul 13 20:59:19.705852 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.705831    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e5ea94-79a2-4799-a1a8-ece9b70c63f3-kube-api-access-5nhcj" (OuterVolumeSpecName: "kube-api-access-5nhcj") pod "13e5ea94-79a2-4799-a1a8-ece9b70c63f3" (UID: "13e5ea94-79a2-4799-a1a8-ece9b70c63f3"). InnerVolumeSpecName "kube-api-access-5nhcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:59:19.804943 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:19.804907    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhcj\" (UniqueName: \"kubernetes.io/projected/13e5ea94-79a2-4799-a1a8-ece9b70c63f3-kube-api-access-5nhcj\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:20.463149 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:20.463121    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:20.463149 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:20.463145    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96d71c1f8e7b2c8bfa1416ecfc50bba32046bfbe3db961a759105b73a3edfda"
Jul 13 20:59:20.473177 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:59:20.473151    2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7326-140510225509232\" is forbidden: User \"system:node:ip-10-0-128-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-62.ec2.internal' and this object" podUID="13e5ea94-79a2-4799-a1a8-ece9b70c63f3" pod="openshift-ingress/test-curl-7326-140510225509232"
Jul 13 20:59:20.636823 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:20.636794    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e5ea94-79a2-4799-a1a8-ece9b70c63f3" path="/var/lib/kubelet/pods/13e5ea94-79a2-4799-a1a8-ece9b70c63f3/volumes"
Jul 13 20:59:22.544367 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.544326    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr" podUID="5c314f52-768e-40bc-bf6c-7fb227b14f90" containerName="main" containerID="cri-o://fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb" gracePeriod=45
Jul 13 20:59:22.792237 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.792217    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:59:22.927759 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.927721    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-home\") pod \"5c314f52-768e-40bc-bf6c-7fb227b14f90\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") "
Jul 13 20:59:22.927910 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.927787    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-model-cache\") pod \"5c314f52-768e-40bc-bf6c-7fb227b14f90\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") "
Jul 13 20:59:22.927910 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.927842    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c314f52-768e-40bc-bf6c-7fb227b14f90-tls-certs\") pod \"5c314f52-768e-40bc-bf6c-7fb227b14f90\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") "
Jul 13 20:59:22.927910 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.927870    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-tmp-dir\") pod \"5c314f52-768e-40bc-bf6c-7fb227b14f90\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") "
Jul 13 20:59:22.927910 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.927898    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-dshm\") pod \"5c314f52-768e-40bc-bf6c-7fb227b14f90\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") "
Jul 13 20:59:22.928107 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.927928    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srnqc\" (UniqueName: \"kubernetes.io/projected/5c314f52-768e-40bc-bf6c-7fb227b14f90-kube-api-access-srnqc\") pod \"5c314f52-768e-40bc-bf6c-7fb227b14f90\" (UID: \"5c314f52-768e-40bc-bf6c-7fb227b14f90\") "
Jul 13 20:59:22.928107 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.928040    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-home" (OuterVolumeSpecName: "home") pod "5c314f52-768e-40bc-bf6c-7fb227b14f90" (UID: "5c314f52-768e-40bc-bf6c-7fb227b14f90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:59:22.928216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.928086    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c314f52-768e-40bc-bf6c-7fb227b14f90" (UID: "5c314f52-768e-40bc-bf6c-7fb227b14f90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:59:22.928216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.928156    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c314f52-768e-40bc-bf6c-7fb227b14f90" (UID: "5c314f52-768e-40bc-bf6c-7fb227b14f90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:59:22.928216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.928182    2584 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-home\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:22.928216 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.928197    2584 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-model-cache\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:22.930061 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.930038    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c314f52-768e-40bc-bf6c-7fb227b14f90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c314f52-768e-40bc-bf6c-7fb227b14f90" (UID: "5c314f52-768e-40bc-bf6c-7fb227b14f90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:59:22.930159 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.930093    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c314f52-768e-40bc-bf6c-7fb227b14f90-kube-api-access-srnqc" (OuterVolumeSpecName: "kube-api-access-srnqc") pod "5c314f52-768e-40bc-bf6c-7fb227b14f90" (UID: "5c314f52-768e-40bc-bf6c-7fb227b14f90"). InnerVolumeSpecName "kube-api-access-srnqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:59:22.930159 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:22.930100    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-dshm" (OuterVolumeSpecName: "dshm") pod "5c314f52-768e-40bc-bf6c-7fb227b14f90" (UID: "5c314f52-768e-40bc-bf6c-7fb227b14f90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:59:23.029050 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.029027    2584 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c314f52-768e-40bc-bf6c-7fb227b14f90-tls-certs\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:23.029050 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.029050    2584 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-tmp-dir\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:23.029193 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.029062    2584 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c314f52-768e-40bc-bf6c-7fb227b14f90-dshm\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:23.029193 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.029077    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srnqc\" (UniqueName: \"kubernetes.io/projected/5c314f52-768e-40bc-bf6c-7fb227b14f90-kube-api-access-srnqc\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:23.479370 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.479339    2584 generic.go:356] "Generic (PLEG): container finished" podID="5c314f52-768e-40bc-bf6c-7fb227b14f90" containerID="fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb" exitCode=0
Jul 13 20:59:23.479580 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.479419    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"
Jul 13 20:59:23.479580 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.479437    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr" event={"ID":"5c314f52-768e-40bc-bf6c-7fb227b14f90","Type":"ContainerDied","Data":"fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb"}
Jul 13 20:59:23.479580 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.479483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr" event={"ID":"5c314f52-768e-40bc-bf6c-7fb227b14f90","Type":"ContainerDied","Data":"de633bbce80c4c10bcd2dde28f59dc53333bf0a558bacd850d30f1356786a957"}
Jul 13 20:59:23.479580 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.479501    2584 scope.go:117] "RemoveContainer" containerID="fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb"
Jul 13 20:59:23.488232 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.488216    2584 scope.go:117] "RemoveContainer" containerID="fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb"
Jul 13 20:59:23.488536 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:59:23.488509    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb\": container with ID starting with fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb not found: ID does not exist" containerID="fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb"
Jul 13 20:59:23.488606 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.488545    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb"} err="failed to get container status \"fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb\": rpc error: code = NotFound desc = could not find container \"fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb\": container with ID starting with fc4725b530fc15ae7edc19315efe494916e5d76231e0feabc56ca3e94fa40ccb not found: ID does not exist"
Jul 13 20:59:23.498907 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.498883    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"]
Jul 13 20:59:23.502829 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:23.502808    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-264ac7/rate-test-model-264ac7-kserve-685c5cdd7-lc4zr"]
Jul 13 20:59:24.636185 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:24.636145    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c314f52-768e-40bc-bf6c-7fb227b14f90" path="/var/lib/kubelet/pods/5c314f52-768e-40bc-bf6c-7fb227b14f90/volumes"
Jul 13 20:59:25.768877 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:25.768841    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:25.820626 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:25.820597    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:35.842621 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:35.842582    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"]
Jul 13 20:59:35.843046 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:35.842829    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" podUID="eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" containerName="maas-api" containerID="cri-o://1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969" gracePeriod=30
Jul 13 20:59:35.866954 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:35.866927    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:36.073677 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.073654    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:59:36.226266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.226183    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-openshift-service-ca\") pod \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") "
Jul 13 20:59:36.226266 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.226251    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrqs6\" (UniqueName: \"kubernetes.io/projected/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-kube-api-access-qrqs6\") pod \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") "
Jul 13 20:59:36.226521 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.226274    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls\") pod \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\" (UID: \"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c\") "
Jul 13 20:59:36.226640 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.226617    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" (UID: "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:59:36.228597 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.228575    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" (UID: "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:59:36.228597 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.228578    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-kube-api-access-qrqs6" (OuterVolumeSpecName: "kube-api-access-qrqs6") pod "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" (UID: "eb64262a-2c3b-4ab6-a764-ef6d9b88c26c"). InnerVolumeSpecName "kube-api-access-qrqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:59:36.327706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.327678    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:36.327706 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.327703    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqs6\" (UniqueName: \"kubernetes.io/projected/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-kube-api-access-qrqs6\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:36.327867 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.327713    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:36.539603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.539512    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" containerID="1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969" exitCode=0
Jul 13 20:59:36.539603 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.539582    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"
Jul 13 20:59:36.539835 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.539605    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" event={"ID":"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c","Type":"ContainerDied","Data":"1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969"}
Jul 13 20:59:36.539835 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.539697    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq" event={"ID":"eb64262a-2c3b-4ab6-a764-ef6d9b88c26c","Type":"ContainerDied","Data":"3f4cf81089f6c8cb118bd7b1f42683b523587e1800dc862ca1b5f53dea3f5f79"}
Jul 13 20:59:36.539835 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.539727    2584 scope.go:117] "RemoveContainer" containerID="1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969"
Jul 13 20:59:36.548940 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.548924    2584 scope.go:117] "RemoveContainer" containerID="1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969"
Jul 13 20:59:36.549176 ip-10-0-128-62 kubenswrapper[2584]: E0713 20:59:36.549156    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969\": container with ID starting with 1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969 not found: ID does not exist" containerID="1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969"
Jul 13 20:59:36.549239 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.549187    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969"} err="failed to get container status \"1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969\": rpc error: code = NotFound desc = could not find container \"1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969\": container with ID starting with 1bbf79b40c5978898b99b6196e2e0057a723cdb5d623816a1367303a6f1c1969 not found: ID does not exist"
Jul 13 20:59:36.560097 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.560077    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"]
Jul 13 20:59:36.563768 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.563744    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-264ac7-67fb85bf8f-k8wvq"]
Jul 13 20:59:36.637660 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:36.637628    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" path="/var/lib/kubelet/pods/eb64262a-2c3b-4ab6-a764-ef6d9b88c26c/volumes"
Jul 13 20:59:40.391143 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:40.391110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:46.236506 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:46.236460    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:46.296779 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:46.296738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:51.100949 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.100916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:51.346341 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.346307    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:51.463160 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463075    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"]
Jul 13 20:59:51.463356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463343    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e5ea94-79a2-4799-a1a8-ece9b70c63f3" containerName="test-curl-7326-140510225509232"
Jul 13 20:59:51.463356 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463357    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e5ea94-79a2-4799-a1a8-ece9b70c63f3" containerName="test-curl-7326-140510225509232"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463367    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c314f52-768e-40bc-bf6c-7fb227b14f90" containerName="main"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463372    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c314f52-768e-40bc-bf6c-7fb227b14f90" containerName="main"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463383    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" containerName="maas-api"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463389    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" containerName="maas-api"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463455    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c314f52-768e-40bc-bf6c-7fb227b14f90" containerName="main"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463461    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e5ea94-79a2-4799-a1a8-ece9b70c63f3" containerName="test-curl-7326-140510225509232"
Jul 13 20:59:51.463490 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.463470    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb64262a-2c3b-4ab6-a764-ef6d9b88c26c" containerName="maas-api"
Jul 13 20:59:51.466242 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.466226    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:51.468770 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.468750    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-84mvh\""
Jul 13 20:59:51.473015 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.472993    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"]
Jul 13 20:59:51.534794 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.534762    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j\" (UID: \"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:51.635461 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.635425    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j\" (UID: \"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:51.635993 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.635971    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j\" (UID: \"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:51.776817 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.776720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:51.918921 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.918882    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"]
Jul 13 20:59:51.921916 ip-10-0-128-62 kubenswrapper[2584]: W0713 20:59:51.921893    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9ce9d7_9f43_4d8a_a8a2_fb1cea942f9d.slice/crio-a863b576168c6e869cce16db4222fc4b16eca2c102e283e15bef46a064092fc6 WatchSource:0}: Error finding container a863b576168c6e869cce16db4222fc4b16eca2c102e283e15bef46a064092fc6: Status 404 returned error can't find the container with id a863b576168c6e869cce16db4222fc4b16eca2c102e283e15bef46a064092fc6
Jul 13 20:59:51.938013 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:51.937974    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:52.039077 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:52.038973    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 20:59:52.609162 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:52.609130    2584 generic.go:356] "Generic (PLEG): container finished" podID="1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d" containerID="1be45f5b2ce11385f335c220867f0c3a494fc7ffed612585e739f67fda8f8890" exitCode=0
Jul 13 20:59:52.609518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:52.609210    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j" event={"ID":"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d","Type":"ContainerDied","Data":"1be45f5b2ce11385f335c220867f0c3a494fc7ffed612585e739f67fda8f8890"}
Jul 13 20:59:52.609518 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:52.609265    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j" event={"ID":"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d","Type":"ContainerStarted","Data":"a863b576168c6e869cce16db4222fc4b16eca2c102e283e15bef46a064092fc6"}
Jul 13 20:59:53.742946 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:53.742920    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:53.751656 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:53.751635    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d-maas-api-service-ca\") pod \"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d\" (UID: \"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d\") "
Jul 13 20:59:53.751932 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:53.751912    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d" (UID: "1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:59:53.852279 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:53.852242    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d-maas-api-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 20:59:54.620573 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:54.620541    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j"
Jul 13 20:59:54.620757 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:54.620542    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j" event={"ID":"1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d","Type":"ContainerDied","Data":"a863b576168c6e869cce16db4222fc4b16eca2c102e283e15bef46a064092fc6"}
Jul 13 20:59:54.620757 ip-10-0-128-62 kubenswrapper[2584]: I0713 20:59:54.620670    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a863b576168c6e869cce16db4222fc4b16eca2c102e283e15bef46a064092fc6"
Jul 13 21:00:01.832508 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:01.832478    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"]
Jul 13 21:00:01.832920 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:01.832708    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv" podUID="eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" containerName="maas-api" containerID="cri-o://8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071" gracePeriod=30
Jul 13 21:00:01.897306 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:01.897268    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:02.276073 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.276050    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 21:00:02.311627 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.311599    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-openshift-service-ca\") pod \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") "
Jul 13 21:00:02.311769 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.311635    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-maas-api-tls\") pod \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") "
Jul 13 21:00:02.311769 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.311692    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4qhx\" (UniqueName: \"kubernetes.io/projected/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-kube-api-access-q4qhx\") pod \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\" (UID: \"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420\") "
Jul 13 21:00:02.312038 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.312011    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" (UID: "eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 21:00:02.313827 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.313797    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" (UID: "eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 21:00:02.313912 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.313830    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-kube-api-access-q4qhx" (OuterVolumeSpecName: "kube-api-access-q4qhx") pod "eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" (UID: "eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420"). InnerVolumeSpecName "kube-api-access-q4qhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 21:00:02.413087 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.413011    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-openshift-service-ca\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 21:00:02.413087 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.413035    2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-maas-api-tls\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 21:00:02.413087 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.413044    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qhx\" (UniqueName: \"kubernetes.io/projected/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420-kube-api-access-q4qhx\") on node \"ip-10-0-128-62.ec2.internal\" DevicePath \"\""
Jul 13 21:00:02.655994 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.655960    2584 generic.go:356] "Generic (PLEG): container finished" podID="eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" containerID="8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071" exitCode=0
Jul 13 21:00:02.656164 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.656023    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"
Jul 13 21:00:02.656164 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.656070    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv" event={"ID":"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420","Type":"ContainerDied","Data":"8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071"}
Jul 13 21:00:02.656164 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.656159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv" event={"ID":"eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420","Type":"ContainerDied","Data":"e37e0da736ac95fe628b70a258c4b22108df24e9dcdd36dd3758feedaa1a7122"}
Jul 13 21:00:02.656341 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.656190    2584 scope.go:117] "RemoveContainer" containerID="8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071"
Jul 13 21:00:02.664411 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.664369    2584 scope.go:117] "RemoveContainer" containerID="8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071"
Jul 13 21:00:02.664671 ip-10-0-128-62 kubenswrapper[2584]: E0713 21:00:02.664652    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071\": container with ID starting with 8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071 not found: ID does not exist" containerID="8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071"
Jul 13 21:00:02.664741 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.664678    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071"} err="failed to get container status \"8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071\": rpc error: code = NotFound desc = could not find container \"8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071\": container with ID starting with 8968f147f03d8dacce7f69a0f741e16ba194242f007dc56c62ac8657ff196071 not found: ID does not exist"
Jul 13 21:00:02.671678 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.671657    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"]
Jul 13 21:00:02.675818 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:02.675797    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-11843d-7b96889fd6-7ddlv"]
Jul 13 21:00:04.637662 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:04.637627    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" path="/var/lib/kubelet/pods/eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420/volumes"
Jul 13 21:00:06.581062 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:06.581023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:12.609296 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:12.609255    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:12.709418 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:12.709365    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:17.273276 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:17.273237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:17.505921 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:17.505883    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:17.960128 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:17.960097    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c9d6b7c9f-rbx8q_7176977e-78c3-4e15-a0db-d2b754cc6994/authorino/0.log"
Jul 13 21:00:28.578059 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:28.578023    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r8xqf_deca5913-494d-4207-8834-c9d128709c3b/manager/0.log"
Jul 13 21:00:28.824688 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:28.824655    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b56dd55f-fb9gq_e15571ba-9b87-47f3-a1d5-c08cefe8a9c6/copy-manifests/0.log"
Jul 13 21:00:28.831315 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:28.831240    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b56dd55f-fb9gq_e15571ba-9b87-47f3-a1d5-c08cefe8a9c6/manager/0.log"
Jul 13 21:00:29.425103 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:29.425071    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732910-2pd7t_6866738a-b5f9-4a8f-af63-4121e155309c/cleanup/0.log"
Jul 13 21:00:29.535853 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:29.535823    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29732925-rr9cv_0cdea227-b6c5-4f66-8004-8d1746aeabc8/cleanup/0.log"
Jul 13 21:00:30.096573 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:30.096540    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-590516-fx6r2_8c816668-21a1-4fbd-8eb9-23f6c03317e7/revoke-keys/0.log"
Jul 13 21:00:30.210346 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:30.210314    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-4c03cf2d-7429n_490644b6-1023-4a8b-8ad2-3c3e5d415eb3/revoke-keys/0.log"
Jul 13 21:00:30.909612 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:30.909582    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-11843d-jzf4j_1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d/revoke-keys/0.log"
Jul 13 21:00:32.342721 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:32.342689    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c9d6b7c9f-rbx8q_7176977e-78c3-4e15-a0db-d2b754cc6994/authorino/0.log"
Jul 13 21:00:32.684193 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:32.684115    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jknvw_fb68c277-61e1-42a4-a595-cd88fc4686fc/kuadrant-console-plugin/0.log"
Jul 13 21:00:33.788654 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:33.788623    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779f44c978-67b7s_2259b5d7-2dd2-47f1-a506-e38a528950c8/kube-auth-proxy/0.log"
Jul 13 21:00:34.873749 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:34.873710    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wdg89_b0067c0d-2a28-4c0b-9c52-b354b794900b/main/0.log"
Jul 13 21:00:35.332785 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:35.332689    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-qz46s_9225f2a5-8be3-475c-aadf-5c44a1b26247/main/0.log"
Jul 13 21:00:39.419220 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419185    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-blmgq/must-gather-mknqt"]
Jul 13 21:00:39.419608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419481    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" containerName="maas-api"
Jul 13 21:00:39.419608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419491    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" containerName="maas-api"
Jul 13 21:00:39.419608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419502    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d" containerName="revoke-keys"
Jul 13 21:00:39.419608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419509    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d" containerName="revoke-keys"
Jul 13 21:00:39.419608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419561    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2b9e5f-d09a-4b6a-be6a-2e5a67cee420" containerName="maas-api"
Jul 13 21:00:39.419608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.419571    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9ce9d7-9f43-4d8a-a8a2-fb1cea942f9d" containerName="revoke-keys"
Jul 13 21:00:39.422443 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.422419    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.422692 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.422658    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:39.425201 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.425178    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-blmgq\"/\"openshift-service-ca.crt\""
Jul 13 21:00:39.426347 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.426328    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-blmgq\"/\"default-dockercfg-8vwhj\""
Jul 13 21:00:39.426450 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.426370    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-blmgq\"/\"kube-root-ca.crt\""
Jul 13 21:00:39.432103 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.432080    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-blmgq/must-gather-mknqt"]
Jul 13 21:00:39.494729 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.494700    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7439e1cc-e10a-470f-b084-67e3fded952e-must-gather-output\") pod \"must-gather-mknqt\" (UID: \"7439e1cc-e10a-470f-b084-67e3fded952e\") " pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.494854 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.494731    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr6n\" (UniqueName: \"kubernetes.io/projected/7439e1cc-e10a-470f-b084-67e3fded952e-kube-api-access-tgr6n\") pod \"must-gather-mknqt\" (UID: \"7439e1cc-e10a-470f-b084-67e3fded952e\") " pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.503275 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.503247    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log" limit=1024
Jul 13 21:00:39.595247 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.595214    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7439e1cc-e10a-470f-b084-67e3fded952e-must-gather-output\") pod \"must-gather-mknqt\" (UID: \"7439e1cc-e10a-470f-b084-67e3fded952e\") " pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.595247 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.595248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr6n\" (UniqueName: \"kubernetes.io/projected/7439e1cc-e10a-470f-b084-67e3fded952e-kube-api-access-tgr6n\") pod \"must-gather-mknqt\" (UID: \"7439e1cc-e10a-470f-b084-67e3fded952e\") " pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.595608 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.595589    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7439e1cc-e10a-470f-b084-67e3fded952e-must-gather-output\") pod \"must-gather-mknqt\" (UID: \"7439e1cc-e10a-470f-b084-67e3fded952e\") " pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.603025 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.602999    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr6n\" (UniqueName: \"kubernetes.io/projected/7439e1cc-e10a-470f-b084-67e3fded952e-kube-api-access-tgr6n\") pod \"must-gather-mknqt\" (UID: \"7439e1cc-e10a-470f-b084-67e3fded952e\") " pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.732043 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.731967    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-blmgq/must-gather-mknqt"
Jul 13 21:00:39.860000 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:39.859971    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-blmgq/must-gather-mknqt"]
Jul 13 21:00:39.862468 ip-10-0-128-62 kubenswrapper[2584]: W0713 21:00:39.862445    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7439e1cc_e10a_470f_b084_67e3fded952e.slice/crio-04a1e5a120b589cb3bbc731ba1c33771bc4a6169a746d8d85dd6112a9c23205a WatchSource:0}: Error finding container 04a1e5a120b589cb3bbc731ba1c33771bc4a6169a746d8d85dd6112a9c23205a: Status 404 returned error can't find the container with id 04a1e5a120b589cb3bbc731ba1c33771bc4a6169a746d8d85dd6112a9c23205a
Jul 13 21:00:40.823542 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:40.823502    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-blmgq/must-gather-mknqt" event={"ID":"7439e1cc-e10a-470f-b084-67e3fded952e","Type":"ContainerStarted","Data":"a85ec03377aade8dc516bd0e88f5c9bcea34b58656033b78cec5823e2b3701ea"}
Jul 13 21:00:40.823948 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:40.823555    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-blmgq/must-gather-mknqt" event={"ID":"7439e1cc-e10a-470f-b084-67e3fded952e","Type":"ContainerStarted","Data":"04a1e5a120b589cb3bbc731ba1c33771bc4a6169a746d8d85dd6112a9c23205a"}
Jul 13 21:00:41.829668 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:41.829633    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-blmgq/must-gather-mknqt" event={"ID":"7439e1cc-e10a-470f-b084-67e3fded952e","Type":"ContainerStarted","Data":"d42af93662842783077ae69665a7fac4a305987cd9a46251df3712564e4506ae"}
Jul 13 21:00:41.844164 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:41.844108    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-blmgq/must-gather-mknqt" podStartSLOduration=2.030570239 podStartE2EDuration="2.844087784s" totalImagesPullingTime="813.517545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:00:41.843308237 +0000 UTC m=+2775.744010906" watchObservedRunningTime="2026-07-13 21:00:41.844087784 +0000 UTC m=+2775.744790447"
Jul 13 21:00:42.260806 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:42.260772    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gvtll_b88f467e-b9f0-403b-a2f4-f08a7980c965/global-pull-secret-syncer/0.log"
Jul 13 21:00:42.313102 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:42.313066    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6s44f_fd8e839d-4afb-4c79-b577-6a8a6b3170ca/konnectivity-agent/0.log"
Jul 13 21:00:42.407034 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:42.406997    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-62.ec2.internal_121fbd70dde46c6dca67ebc08b171187/haproxy/0.log"
Jul 13 21:00:47.323639 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:47.322546    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-c9d6b7c9f-rbx8q_7176977e-78c3-4e15-a0db-d2b754cc6994/authorino/0.log"
Jul 13 21:00:47.423045 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:47.423013    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jknvw_fb68c277-61e1-42a4-a595-cd88fc4686fc/kuadrant-console-plugin/0.log"
Jul 13 21:00:49.653792 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:49.653757    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6nr4c_eac57dfb-d773-40de-bf82-430dcc3601f4/node-exporter/0.log"
Jul 13 21:00:49.685648 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:49.685620    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6nr4c_eac57dfb-d773-40de-bf82-430dcc3601f4/kube-rbac-proxy/0.log"
Jul 13 21:00:49.709730 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:49.709696    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6nr4c_eac57dfb-d773-40de-bf82-430dcc3601f4/init-textfile/0.log"
Jul 13 21:00:53.006783 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.006747    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"]
Jul 13 21:00:53.011632 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.011600    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.020288 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.018862    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"]
Jul 13 21:00:53.125349 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.125304    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-sys\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.125349 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.125337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-proc\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.125619 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.125366    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-podres\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.125619 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.125516    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-lib-modules\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.125619 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.125552    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9rt\" (UniqueName: \"kubernetes.io/projected/04bfb7db-3519-4771-954f-4a30e2a1e329-kube-api-access-dk9rt\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226106 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226073    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-sys\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226106 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226108    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-proc\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226299 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-podres\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226299 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226175    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-lib-modules\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226299 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226194    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9rt\" (UniqueName: \"kubernetes.io/projected/04bfb7db-3519-4771-954f-4a30e2a1e329-kube-api-access-dk9rt\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226299 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226204    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-sys\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226299 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226215    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-proc\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226491 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226326    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-lib-modules\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.226491 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.226325    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04bfb7db-3519-4771-954f-4a30e2a1e329-podres\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.235096 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.235058    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9rt\" (UniqueName: \"kubernetes.io/projected/04bfb7db-3519-4771-954f-4a30e2a1e329-kube-api-access-dk9rt\") pod \"perf-node-gather-daemonset-k8wtr\" (UID: \"04bfb7db-3519-4771-954f-4a30e2a1e329\") " pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.326091 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.326011    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.490186 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.489608    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"]
Jul 13 21:00:53.920036 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.919503    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr" event={"ID":"04bfb7db-3519-4771-954f-4a30e2a1e329","Type":"ContainerStarted","Data":"d71b24b0512bb6d1b7b74953b99927137588c331e3169adff30bbcd3fb4d16e8"}
Jul 13 21:00:53.920036 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.919568    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr" event={"ID":"04bfb7db-3519-4771-954f-4a30e2a1e329","Type":"ContainerStarted","Data":"d3073b375731fe22767c3b9a9051003aafa57853d9626006328699096c3c0e92"}
Jul 13 21:00:53.920036 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.919910    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:00:53.936822 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:53.936761    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr" podStartSLOduration=1.936739823 podStartE2EDuration="1.936739823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:00:53.935212282 +0000 UTC m=+2787.835914945" watchObservedRunningTime="2026-07-13 21:00:53.936739823 +0000 UTC m=+2787.837442486"
Jul 13 21:00:54.527751 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:54.527712    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wd8gn_74bd3172-a17f-4910-b38c-c79719d75df4/dns/0.log"
Jul 13 21:00:54.548978 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:54.548940    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wd8gn_74bd3172-a17f-4910-b38c-c79719d75df4/kube-rbac-proxy/0.log"
Jul 13 21:00:54.569682 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:54.569655    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4nkxs_33bbf7c8-d5f1-4bfe-bbe4-75eaa3829d34/dns-node-resolver/0.log"
Jul 13 21:00:55.123461 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:55.123433    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6647c786d4-5x7qc_02e6065a-a51f-4f12-9e99-77482bc178a6/registry/0.log"
Jul 13 21:00:55.145601 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:55.145572    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4h87g_2efc0e75-f975-4046-b42f-c142d7ed511b/node-ca/0.log"
Jul 13 21:00:56.219228 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:56.219189    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779f44c978-67b7s_2259b5d7-2dd2-47f1-a506-e38a528950c8/kube-auth-proxy/0.log"
Jul 13 21:00:57.113717 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:57.113673    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qncsw_66f0e938-3e7f-4e3b-852f-b6998d68711f/serve-healthcheck-canary/0.log"
Jul 13 21:00:57.739676 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:57.739649    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fww5j_ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb/kube-rbac-proxy/0.log"
Jul 13 21:00:57.758785 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:57.758759    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fww5j_ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb/exporter/0.log"
Jul 13 21:00:57.778386 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:57.778353    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fww5j_ffd7ce5f-3c9d-4e8a-a96a-c3db6d94fbfb/extractor/0.log"
Jul 13 21:00:59.935233 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:00:59.935204    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-blmgq/perf-node-gather-daemonset-k8wtr"
Jul 13 21:01:00.315461 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:00.315362    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r8xqf_deca5913-494d-4207-8834-c9d128709c3b/manager/0.log"
Jul 13 21:01:00.420070 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:00.420039    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b56dd55f-fb9gq_e15571ba-9b87-47f3-a1d5-c08cefe8a9c6/manager/0.log"
Jul 13 21:01:00.437977 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:00.437944    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b56dd55f-fb9gq_e15571ba-9b87-47f3-a1d5-c08cefe8a9c6/copy-manifests/0.log"
Jul 13 21:01:01.714828 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:01.714794    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f676ff564-m2dgv_ac0bacd5-a23d-4e9a-9606-96eb0fe92948/manager/0.log"
Jul 13 21:01:02.698808 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:02.698778    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779f44c978-67b7s_2259b5d7-2dd2-47f1-a506-e38a528950c8/kube-auth-proxy/0.log"
Jul 13 21:01:04.183438 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:04.183347    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779f44c978-67b7s_2259b5d7-2dd2-47f1-a506-e38a528950c8/kube-auth-proxy/0.log"
Jul 13 21:01:08.787547 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:08.787518    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/kube-multus-additional-cni-plugins/0.log"
Jul 13 21:01:08.826597 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:08.826562    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/egress-router-binary-copy/0.log"
Jul 13 21:01:08.857467 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:08.857443    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/cni-plugins/0.log"
Jul 13 21:01:08.893465 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:08.893435    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/bond-cni-plugin/0.log"
Jul 13 21:01:08.931445 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:08.931421    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/routeoverride-cni/0.log"
Jul 13 21:01:08.973388 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:08.973361    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/whereabouts-cni-bincopy/0.log"
Jul 13 21:01:09.011437 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:09.011415    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wdwzm_45a5d5fc-dc3a-4d65-b776-21e0697f615d/whereabouts-cni/0.log"
Jul 13 21:01:09.107579 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:09.107549    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-htbsc_98a95222-55d4-4845-a53a-d00cecd8f210/kube-multus/0.log"
Jul 13 21:01:09.139821 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:09.139795    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2vxjq_db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0/network-metrics-daemon/0.log"
Jul 13 21:01:09.173562 ip-10-0-128-62 kubenswrapper[2584]: I0713 21:01:09.173535    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2vxjq_db3c0356-91d7-4b9c-9cbd-5d06cdbedbc0/kube-rbac-proxy/0.log"